ELEMENTS OF BEHAVIOR CHANGE AND ENERGY EFFICIENCY PROGRAMS TRAINING

Transcription

1 ELEMENTS OF BEHAVIOR CHANGE AND ENERGY EFFICIENCY PROGRAMS TRAINING

2 Welcome Welcome to training! Day One: Understanding Behavior and How to Effect Change Day Two: In-depth Planning and Evaluation

3 Agenda Day One: Understanding Behavior and How to Effect Change Introduction Defining Behavior: What is Behavior? Behavior Change Theories Behavior Change Interventions Behavioral Program Types Behavior Strategies in All Programs Key Planning Concepts Implementation

4 Agenda Day One: Understanding Behavior and How to Effect Change Introduction Defining Behavior: What is Behavior? Behavior Change Theories Behavior Change Interventions Behavioral Program Types Behavior Strategies in All Programs Key Planning Concepts Implementation

5 Agenda Day Two: In-depth Planning and Evaluation Designing Behavior with Key Planning Concepts in Mind Evaluation Design Taking the Long View Wrap Up

6 INTRODUCTION THE VALUE OF BEHAVIOR: WHY STUDY BEHAVIOR?

7 DEFINING BEHAVIOR: WHAT IS BEHAVIOR?

8 Agenda Day One: Understanding Behavior and How to Effect Change Introduction Defining Behavior: What is Behavior? Behavior Change Theories Behavior Change Interventions Behavioral Program Types Behavior Strategies in All Programs Key Planning Concepts Implementation

9 Think About 3 Days Ago Think about your day from when you woke up to when you went to sleep. What little behaviors did you do? What big decisions did you have to make? How much of the day do you remember?

10 Activity: What is your definition of behavior?

11 The way in which an animal or person acts in response to a particular situation or stimulus. So... basically everything we do is behavior.

12 Behavior Change Means Lots of Things Decision making Investments Routine purchases Habits Large purchases Behavior Change

13 Behavior Change in Energy Decision making Investments Invest in your home Maintain your home better Use existing equipment better Purchasing habits Large purchases Upgrade your home Automate your home Habits Change how/when you do things Get rid of things Behavior Change

14 Behavior Change and Consumption Looking at this graph, we think of widget programs. But what about... Routine purchases... Cooking 4% Electronics 8% Computers 2% Wet Cleaning 5% Other 5% Space Heating 31% Habits... Decision Making... Refrigeration 6% Investments... These are places where you can change behavior! Lighting 10% Water Heating 14% Space Cooling 15%

15 Interventions... Target one or more specific behaviors Are rooted in social science research Consciously consider which behavior(s) they will affect Yield evaluable results

16 Group Discussion: The many types of behaviors specific to energy!

17 Classifying Behaviors Some behaviors share characteristics with others.

18 Reviewing Classifications: Round-Robin

19 BEHAVIOR CHANGE THEORIES

20 Agenda Day One: Understanding Behavior and How to Effect Change Introduction Defining Behavior: What is Behavior? Behavior Change Theories Behavior Change Interventions Behavioral Program Types Behavior Strategies in All Programs Key Planning Concepts Implementation

21 Theory! Individual, Social, and Science and Technology

22 There s More than Social Norms!

23 Why Theory? Explanatory Theory Instructive Evaluative

24 Individual Theories Levels of Theory

25 Social Theories Levels of Theory

26 Science and Technology Studies Levels of Theory

27 Cognitive Psychology Priming sets the stage for a desired action by introducing customers to information or suggestion prior to the point of decision. Individual Theories

28 Cognitive Psychology Framing is a way to provide energy information in a manner that takes advantage of existing biases or understandings. Individual Theories

29 Social Psychology Values, Beliefs, Norms Theory suggests that individual begins with personal values which leads to beliefs about behavior and norms. Individual Theories

30 Behavioral Economics Anchoring Describes how the first piece of information offered to us affects our perceptions about later pieces of information. Individual Theories

31 Sociology Life Choices and Life Chances suggests that how we behave is determined by both our choices and our circumstances/environment. Social Theories

32 Sociology Reference Groups: Individuals and groups judge their behavior by their reference groups. These may be the groups of which they are a part, or can be groups to which they aspire. Social Theories

33 Science and Technology Studies Material Satisfaction Model suggests that consumers are happy when product features meet their needs. Emotional Satisfaction Model suggests that satisfaction depends on the material features and how customers relate to the product based on their expectations and environment. Science and Technology

34 Activity: Theory What is it Good For?

35 Agenda Day One: Understanding Behavior and How to Effect Change Introduction Defining Behavior: What is Behavior? Behavior Change Theories Behavior Change Interventions Behavioral Program Types Behavior Strategies in All Programs Key Planning Concepts Implementation

36 BEHAVIOR CHANGE INTERVENTIONS

37 Quick Round Robin What do you think is a behavior intervention? Are there any interventions you can think of that support your definition?

38 Review of Behavior Interventions Traditional: Financial Incentive Energy Pricing Leveraging Sunk Cost Legal Information Underused: Commitment Feedback Follow-through Framing In-person Interactions Rewards or Gifts Social Norms Stacked Interventions

39 Levels of Theory Individual Theories Social Theories Science and Technology Studies

40 Commitment

41 Public Commitment When people make a public commitment they are more likely to change a behavior. Commitments are most powerful when used with other strategies. For example, feedback or norming.

42 Commitment Example: 20/20 Challenge People asked to bring old lights to exchange for CFLs Then, people asked to pledge to reduce their energy use Track effectiveness of the pledge Reported reduction Asking for a small action Commitment 20/20 Challenge lacked this component 20/20 Challenge, FortisBC

43 Feedback

44 Feedback: A Promising Strategy Enhanced Billing Household specific info, advice Estimated Feedback Web-based energy audits with info on an ongoing basis Daily/Weekly Feedback Household specific info, advise on daily or weekly basis "Indirect" Feedback (Provided after Consumption Occurs) Real-Time Feedback Real-time premise level info Real-Time Plus Feedback Real-time info down to the appliance level "Direct" Feedback (Provided in Real Time) Higher Savings Lower Savings Lower Cost Higher Cost

45 Feedback Example: HERs ~1-2% Savings

46 Follow-through Goal setting Prompts

47 Goal Setting Individuals must be committed to achieve the goal. Goals should be clear and reachable. Works well if it is paired with another strategy such as public commitment or feedback. The more challenging the goal, the more important goal commitment becomes.

48 Prompts Prompts are... Timely Located near place where an action occurs Targets a specific behavior

49 Framing Setting the default action Focus on losses rather than gains

50 Framing Helps Shift Perspectives

51 Set the Default Action Examples Love to stick with what we know Habit Thermostats TV Auto-Brightness Ideal for habits or simple choices. Avoid when choice is difficult. Employee Savings Plans Peak Day Pricing

52 Frame Losses How much is this bottle worth? Now would you accept $2 for the water?

53 In-person Interactions

54 The Power of In-Person Interaction Also known as personal contact or person-to-person interaction. Explains community-based marketing efforts. Credibility and trust highly important. Strategies that involve engaging the efficiency advocates or community leaders to deliver the message.

55 Rewards or Gifts

56 Rewards or Gifts: Many Strategies Offer gifts (e.g., free LED bulbs) Offer a chance to win a larger reward (lottery) Offer a chance to win a prize (competition)

57 Social Norms

58 We Like to be in the Norm Um

59 Make the Target Behavior Seem Normal Helps motivate people to do an action. Impactful for those that don t care as much. When the norm is NOT your target behavior: Focus on approval of behavior. If you can t avoid using social norms will only serve to demotivate people.

60 Social norms: Example

61 Stacked Interventions

62 Why we Stack 100% Percent Adoption The combination can have a larger impact than either on their own. 0% Time Commitment Social Norms Commitment and Social Norms

63 When you Stack Interventions... Combinations need to be Selective Thoughtful

64 Example of a Simple Stack Provided Prompts The Problem! Changed Defaults Default: ON Default: OFF No Prompt Prompt 3% 11% 15% 48%

65 Intervention Playing Cards In groups: Identify interventions or combination of interventions that address your target behavior to change.

66 Round Robin: Your chosen interventions

67 BEHAVIORAL PROGRAM TYPES

68 Agenda Day One: Understanding Behavior and How to Effect Change Introduction Defining Behavior: What is Behavior? Behavior Change Theories Behavior Change Interventions Behavioral Program Types Behavior Strategies in All Programs Key Planning Concepts Implementation

69 Why Taxonomies? Use the Taxonomy for Clarity Using the taxonomy makes results more easily comparable across different program efforts. This helps build the dataset for behavior-based programs generally. Using the taxonomy with the three family structure aids in designing programs that stack. Source:

70 Three Core Families Three Families of Behavior Programs Cognition Programs where intrinsic psychological processes are foremost Calculus Programs where the deliberation of extrinsic aspects play a primary motivating role Social Interaction Programs whose key drivers are sociability and belonging Source:

71 With Sub Program Areas Taxonomy of Utility Run Behavior Programs COGNITION CALCULUS SOCIAL INTERACTION Communication Efforts Feedback Human Scale General Real Time Community-based Social Marketing Cable or broadcast TV Asynchronous Person-to-person Radio Games Peer Champions Billboards Competitions Eco Teams Other traditional Media Outlets Challenges Online Forums Targeted Lotteries Gifts Enhanced Billing Incentives Direct Mail Cash Bill Inserts Rebates User Friendly Bill Designs Subsidies Social Media Home Energy Audits Facebook Audit Only Twitter Audit + Tumbir Installation Blogs Direct Install Education & Training DIY In Schools K-12 Higher Education In Companies or Institutions Commercial Industrial Source:

72 Cognition Cognition: Motivated by Intrinsic Factors Communication Efforts General Cable or Broadcast TV Radio Billboards Other traditional Media Outlets Social Media Facebook Twitter Tumbir Blogs Targeted Education & Training In Schools Enhanced Billing Direct Mail K-12 Bill Inserts Higher Education User-Friendly Bill Designs In Companies or Institutions Commercial Industrial Source:

73 Cognition

74 Calculus Calculus: Motivated by Extrinsic Factors Feedback Real-Time Asynchronous Incentives Cash Rebates Subsidies Games Competitions Audits Only Challenges Audits + Lotteries Installation Direct Install DIY Home Energy Audits Source:

75 Calculus Examples

76 Social Interactions Social Interaction: Sociability and Belonging Human Scale Community-based Social Marketing Person-to-person Peer Champions Eco Teams Online Forums Gifts Source:

77 Social Interactions

78 Behavior Program Jeopardy!

79 BEHAVIOR STRATEGIES IN ALL PROGRAMS

80 Agenda Day One: Understanding Behavior and How to Effect Change Introduction Defining Behavior: What is Behavior? Behavior Change Theories Behavior Change Interventions Behavioral Program Types Behavior Strategies in All Programs Key Planning Concepts Implementation

81 Behavior in all Programs Behavior is not just for behavior programs. Behavioral interventions can be integrated into all programs, including mature programs.

82 Stacking Interventions Stacking can create more effective programs. Frame communications for loss aversion Set socket goals to prompt greater replacement rates Provide feedback on goal attainment

83 Populus Calculus and Social Interactions

84 Smart Energy Now

85 Peak Day Pricing Support

86 National Grid RI Statewide Program Rewards RED Rewards RED Rewards RED Electric HER Electric Control Dual Fuel HER DF Control Gas HER Gas Control New Movers Control Thermostats Statewide Community Outreach Statewide online web access to feedback Model of National Grid Rhode Island Statewide Behavior Program

87 CA Emerging Technology Program

88 KEY PLANNING CONCEPTS

89 Agenda Day One: Understanding Behavior and How to Effect Change Introduction Defining Behavior: What is Behavior? Behavior Change Theories Behavior Change Interventions Behavioral Program Types Behavior Strategies in All Programs Key Planning Concepts Implementation

90 Behavior as Part of Design Program Design Behavior Other Components Implementation If behavior is a major component it must be implicit in design, and planned for.

91 Traditional vs. Behavior

92 Key Planning Inputs Measure life Incremental installed cost Cost above baseline equipment Incentive (rebate) Annual energy savings (at meter) Peak demand reduction (at meter) Program cost estimates Behavior components Developed cost of conserved energy and benefit cost analysis

93 DSM Planning Framework Framework DSM Objectives Policies Operational Load Shapes Portfolio Design End Uses Technologies Mktg Techniques Program Design Customer Issues Utility Issues Benefit/Cost EM&V Process & Impact Evaluation Measurement & Verification Information for Feedback Program Implementation Experiments Plot Programs Full-Scale Programs

94 Approach Varies by Type of Program Resource Acquisition Short term analysis and results May vary by type of measure Pipe wrap vs. thermostats Tie with other programs Market Transformation Long term analysis and results Change should be identified in barrier changes using theory Tie with other programs Know your goals before you begin design

95 Long Term Outcomes Near Term Outputs Activities Strategies Barriers Channel Initiatives Lighting, HVAC, Motors, Kitchens Contractors & Distributors Don t Sell Efficiency because they fear losing the sale and they lack understanding of EE benefits, have insufficient sales tools and staff training on EE sales Distributors & Contractors Don t Have EE Units in Stock due to lower demand Customers Don t Demand or Value Efficiency due to lack of understanding, risk of new tech, inadequate info, higher first costs, competitive bid requirements Customers Do Not Perceive Efficient Options Being Worth the Price Difference Channel Support Sales training to both C&D on benefits of EE equipment plus training on strategies for selling those benefits Sales tools for C (sell sheets, calculators) Help C&D understand & promote EE as a value proposition and differentiator Availability of Service Buydown incentives, which help contractors build EE business Channel Stocking Partnerships Monitor and document C&D stocking practices via visits Showcase partners who change their stocking practices in publications Encourage changes in stocking practices via presentations, incentives, and coop advertising Information, Education Sales sheets on benefits, life cycle costs provided by Focus and Contractors Case studies and press releases about overall benefits Increase awareness through presentations at conferences Co-marketing programs Incentives Standard incentives Bonus incentives Help customers understand full value of EE through better sales materials Focus provides third party credibility to sales efforts by market providers Trained C&D understand benefits of EE and use of sales tools to sell efficiency Increased efficient equipment in stock More information available to Incentives spur increased sales of customers & change in understanding of efficient product value of EE Participating Market Providers: Have high level of EE equipment in stock Promote EE as regular part of their offers Adjust their business models to differentiate with EE See profit in promoting energy efficiency Promote EE in their advertising All of which prompts increased market push for EE Customers: Are knowledgeable about EE options Understand the value of EE Consistently invests in efficiency All of which prompts increased market demand for EE products

96 Summary of Program Design Steps Understand the goals both primary and secondary Determine the achievable potential savings Identify markets/channels Understand their barriers and strategy for overcoming them short and long term Develop theory roadmap to get there Identify: Strategies, steps, products, services Behavior and support needed to address barriers Develop into programs Develop program plans Bundle programs into portfolios

97 IMPLEMENTATION

98 Agenda Day One: Understanding Behavior and How to Effect Change Introduction Defining Behavior: What is Behavior? Behavior Change Theories Behavior Change Interventions Behavioral Program Types Behavior Strategies in All Programs Key Planning Concepts Implementation

99 Regulatory Issues Regulatory Issues Must Be Addressed How are savings counted? Are there rewards and incentives for goals? Based on what? Gross, deemed, evaluations, other? How do behavior programs get counted? What about persistence? Experimental Design and Quasi-Experimental Design Can those be addressed in program? Risk Aversion using experiments and can it fit with goals Often times staff is uneducated on behavior programs and experimental design.

100 Attribution Savings Attribution Predetermined baseline Comparison group Control group Observable unique effects Free rider determination decided Double counting can be a problem Issue of who gets credit Spillover determination still being discussed

101 Equipment Focused Programs Equipment Programs with Behavioral Elements Should the behavior be measured separately? It depends on its significance and if it can be measured. Are there both equipment only participants and equipment/behavior participants? If so, you will need to measure and track separately if the behavior difference is seen as significant. Do we see any differences in persistence of savings with the behavioral element? There may be differences on effective savings life which would be important. Can we get spillover savings within the program, or to other programs from behavior addition? If yes, it should be measured and tracked.

102 Costs vs Savings What will be the behavior tracking and evaluation cost versus the savings differential? How savings are computed and credited must be worked out. Tracking behavior savings separately may or may not be worth the costs. Should be worked out ahead of launch.

103 Follow the Money (Savings) Issue is who gets the credit? As programs broaden behavior aspects, giving credit for savings can get more difficult. Actions overlap and the exact stimulus for the attribution of savings may not be clear. For example, a behavior type program giving reports and education may lead to a purchase through an equipment program. However, the equipment program had to be set up and functional for the customer to be able to purchase. Spillover effects can be within a program or with other programs. Double counting savings is important to avoid.

104 Multifaceted Program Credit Multifaceted programs are also a challenge A program may work with both customers and trade allies and have behavior aspects to each. Should you try to break apart credit between those behavior elements or keep within the program? It depends. Size Behavior impacts Whether there are non-behavior options Costs to track and evaluate Whether there is difference in credit or other spillover

105 It s in the Contract Giving Credit Credit for accomplishment means real dollars Utility incentives for savings. More complex is the credit between implementation contractors when there is overlap and spillover between programs. Performance in contractors contract Not in their self interest to give credit to others or have spillover Need to see how to get the contract language to recognize and reward spillover whether behavior driven or driven by other program actions This credit issue can be between gas and electric utilities as well when doing joint programs.

106 Tracking Tracking is often an Achilles heel If it is not tracked, it is hard to evaluate later and costly to get data. Must be consistent with regulatory reporting requirements. Depending on the experimental approach, the customer records for selecting participants will be important. Also critical to track participation across programs so that you can see the spillover effects.

107 Conclusion Implementation issues should be considered during the design of the program so that you end up with something that can be accomplished. Implementation should consider the following throughout program delivery: Regulatory issues, expectations and constraints Cost of savings Attribution of savings Source of savings (behavior or equipment programs?) Program complexity Tracking

108 DESIGING BEHAVIOR WITH KEY PLANNING CONCEPTS IN MIND INTERACTIVE LESSON

109 Agenda Day Two: In-depth Planning and Evaluation Designing Behavior with Key Planning Concepts in Mind Evaluation Design Taking the Long View Wrap Up

110 Design The Foundation for Success Target participants Targeted actions Conservation behaviors? Measures? Both? Type of Interventions Participation design After each main section you will make decisions for your own program Opt-in? Opt-out? Anticipated savings Annual energy savings (at meter) Peak demand reduction (at meter) Designing for evaluation

111 Target Participants Better Targets = Better Results Sectors Sub Groups High Users Commercial Offices Faith Institutions ETC!!!

112 Target Actions Designers can implement pure behavior programs, such as HERs or IHDs, or use integrated approaches where behavior mechanisms can supplement or add-on to existing program models.

113 Target Actions The types of actions you target can impact your savings. Every action has different first year and lifetime savings (and costs). Behaviors habituated) (Not Habituated Behaviors Installed Measures T 0 T 1 T 2 T 3 T 4....

114 Selecting Interventions With a focus on barriers and desired outcomes, carefully select your interventions. What are the primary lifestyle circumstances that prevent people from taking this action? Are there physical, regulatory, or legal barriers to taking action? What is the best way to communicate with customers? Are there better ways to message to them?

115 Participation Type Participation Levels Vary Based on Approach Opt-In programs are programs where customers self select in to the program. Opt-Out programs are programs where customers are assigned to a program. If customers do not want to participate in the program, to which they have been assigned, they can choose to leave the program.

116 Participation Levels Vary Based on Approach Participation Type Opt-In more savings per participant, fewer participants Opt-Out less savings per participant, more participants MASSACHUSETTS THREE YEAR CROSS-CUTTING BEHAVIORAL PROGRAM EVALUATION INTEGRATED REPORT, JULY 2012

117 Anticipated Savings The sources of savings can impact both first year and lifetime goals. MASSACHUSETTS THREE YEAR CROSS-CUTTING BEHAVIORAL PROGRAM EVALUATION INTEGRATED REPORT, JULY 2011

118 Designing for Evaluation As a designer and planner, you have to know how you will be evaluated and plan or design for it! Preponderance of Evidence Designed Before Launch Designed After Launch Opt-Out Opt-In Randomized Control Trials Random Encour. Design Recruit and Delay or Deny Interrupted Time Series Matched Comp. Group Variation in Adoption Pre-post Analysis

119 EVALUATION DESIGN

120 Agenda Day Two: In-depth Planning and Evaluation Designing Behavior with Key Planning Concepts in Mind Evaluation Design Taking the Long View Wrap Up

121 Teasing our Impacts Your primary job as an evaluator is to identify this (a counterfactual). And measure the difference here. Attributable to the program

122 and then tease out the sources of the effects. Lift in participant uptake above counterfactual program influence Took action without other programs direct effects Uptake in non-behavioral programs joint effects (double-counted) Attributable to the program

123 Designing for Evaluation Each evaluation design strategy has its pros and cons, but every one aims to tease out program effects. Preponderance of Evidence Designed Before Launch Designed After Launch Opt-Out Opt-In Randomized Control Trials Random Encour. Design Recruit and Delay or Deny Interrupted Time Series Matched Comp. Group Variation in Adoption Pre-post Analysis

124 Overview of Methods: Experimental Method Description What it Measures Pros and Cons Random Control Design Individuals are randomly allocated to received a program or no treatment (control group). The difference between the two groups. Pro: The gold standard for evaluation. Con: Requires significant amount of control to run well. Favors simple designs. Random Encouragement Design Individuals are randomly assigned to a treatment or a control group. The treatment group is encouraged to participate in the program. The savings effects of encouragement across the entire treatment group. It DOES NOT measure the individual savings associated with individual households that opt-in to the program. Pro: Experimental and highly defensible of methods for attributing effects to a program. Con: Requires a huge sample size to treat enough households to measure an effect. Recruit and Delay or Deny Individuals who opted-in to the study are randomly assigned to a treatment and control group. The treatment group receives treatment. The control group is either denied treatment or treatment is delayed. The savings effects of participation among those who opt-in and receive treatment, compared to those who opted-in but did not receive the treatment. Pro: Experimental and highly defensible of methods for attributing effects to a program. Con: Delaying or denying a treatment may not comply with equity mandates. Con: Results are sample specific in many cases. Con: Does not account for other factors that affect savings. Designed Before Launch

125 Designed After Launch Overview of Methods: Quasi-experimental

126 Random Control Design Design: Measurement: A randomized control trial (RCT) is a specific type of scientific experiment used to test the effectiveness of a program. Target customers are randomly allocated to receive a program or no treatment (control group). The program effect is measured as the difference between the two groups. Bias and limitation: This approach requires a significant amount of control of fielding. Favors simple designs (e.g. reports).

127 Random Control Design Gets program Treatment Target group Randomly assigned to: Control

128 Random Encouragement Design Design: Measurement: Individuals are randomly assigned to a treatment group or control group. The treatment group receives encouragement to participate in the treatment. The control group does not receive encouragement. The treatment group is compared to the control group to measure the savings effects of encouragement across the entire treatment group. This approach does not measure the effects of treatment directly. Bias and limitation: This approach requires large sample sizes and is difficult to implement in real-world settings.

129 Random Encouragement Design Savings effects of encouragement Target group Population Randomly Opted-in assigned to: Treatment group: Receives encouragement to participate Control group: Does not receive encouragement

130 Recruit and Delay or Deny Design: Measurement: Sample of individuals from target population opt to participate in the treatment program. Individuals that opted-in are randomly assigned to a treatment or control group. The treatment group immediately receives the treatment. The control group receives the treatment later (delay) or not at all (deny). Savings effects of the treatment are measured by comparing the effects for treatment and control groups. The savings effects may be attributed to participation among those who opted-in and received treatment, compared to those who opted-in, but did not receive the treatment. Bias and limitations: This approach does not account for a self-selection bias in which the characteristics of the individuals that opted-in also affect the treatment effects.

131 Recruit and Delay or Deny Savings effects of program Treatment group: Receives program Target group Population opted-in Randomly assigned to: Control group: Program is delayed or denied

132 Matched Comparison Group Design: Measurement: Limitations: The treatment group consists of individuals from the target population that opted to participate in the treatment program. The control group is constructed by selecting non-participants that are similar to participants. Similarity is based on one or more observable variables that are likely associated with treatment effects. Variables are measured over the same time period, prior to treatment. Two forms of matching: One-to-One: Individual participants are matched to individual non-participants. Balanced group: The participant group is matched to the non-participant group. Savings effects of the treatment are measured by comparing the treatment group and control groups, controlling for relevant observable characteristics. Assume that the matching variables reflect relevant characteristics.

133 Matched Comparison Group Savings effects of program Treatment group: Receives program Target group Population opted-in Treatment and control groups matched on observable characteristics Control group: Does not receive program

134 Matched Comparison Group

135 Variation in Adoption Design: Measurement: Limitations: A sample of individuals from the target population opt to participate in the treatment program. Individuals that opted-in are randomly assigned to an early treatment group or late treatment (control) group. The early treatment group receives the treatment group immediately. The late treatment group receives the treatment at a later point in time. Usage and conditions before participation of late participants are used as a control group for early participants. Comparisons and matching between early and late participants are done at regular intervals (i.e., monthly) to control for biases and participation propensities. The savings effects are measured by comparing the effects for earlier treatment group to the late treatment groups, before they received the treatment. This approach assumes that there are no differences between early and late participants.

136 Variation in Adoption

137 Preponderance of Evidence Surveys Engineering Observations

138 Evaluating Stacked Program Multi-pronged and stacked programs require careful evaluation design and planning. Rewards RED Rewards RED Rewards RED Electric HER Electric Control Dual Fuel HER DF Control Gas HER Gas Control New Movers Control Thermostats Statewide Community Outreach Statewide online web access to feedback Model of National Grid Rhode Island Statewide Behavior Program

139 Trade-offs in Evaluation Designs There are trade offs in every design decision. Experiments: Certainty in evaluation Other Evaluations: Flexibility in field and design Experiments: Highly controlled fielding and design Other Evaluations: Lack of predictability in measurement

140 Removing other Program Effects Lift in participant uptake above counterfactual program influence Took action without other programs direct effects Uptake in non-behavioral programs joint effects (double-counted) Attributable to the program

141 Avoiding Double-Counting Programs can have a wide range of effects, some might be double-counted. Receives Treatment Behavior outside of other program (a) Measure outside of other program (b) Measure via other programs (c)

142 Removing other Program Effects Eliminating Double Counting Channeled Savings Unique Savings Detected by Billing Analysis Observed Consumption Observed Consumption Control Treatment Overall height of bars represents expected consumption in absence of any utility intervention. We d expect the same consumption in the post-period for treatment and control without the program.

143 Exercise Evaluation Design

144 TAKING THE LONG VIEW

145 Agenda Day Two: In-depth Planning and Evaluation Designing Behavior with Key Planning Concepts in Mind Evaluation Design Taking the Long View Wrap Up

146 Let s think about persistence Traditionally you install a widget...

147 Traditional Definition of Persistence: 1. Measure is installed correctly and saves energy for more than year one. 2. Energy savings maintain after installation without changes in operations or equipment to cause degradation of savings.

148 But even in widget land behavior often interferes with this traditional view!

149 Behavioral Definition of Persistence: 1. Target behavior has been habituated. 2. Energy savings maintain after habit is in place.

150 Two Types of Persistence Persistence with treatment: Maintain behavior but need intermittent reinforcement Persistence without treatment: Maintain behavior without need for any reinforcement interventions

151 Designing to Test Persistence About 2% savings

152 Opower Field Experiments Utilities created a series of experiments testing two hypotheses using Opower s Home Energy Reports: Monthly reports will yield more savings than quarterly reports. Savings estimates will diminish once the utility stops sending out reports.

153 Opower Field Experiments First hypothesis: True. Monthly reports (in red) yield higher savings than quarterly reports.

154 Opower Field Experiments Second hypothesis: True. Households that still receive the report continue to save energy while those that do not save less.

155 Integrating Persistence into Opower Experiments Testing for Persistence Had enough sample to divide up Developed clear, simple hypotheses Created a more complex experiment Implementer and utility worked to allow for future data collection Patience!

156 When to Test for Persistence Ask yourself... How often do treatments need to occur to continue seeing measurable savings? Is there a point at which treatments can stop altogether? For how long? Is the intervention effective? Opower: First documented that their interventions saved energy. Then tested boundaries of the interventions effectiveness.

157 Persistence Across Different Study Designs Opt in: Participants take action to be included in the study. Pros: More motivation, more persistence Cons: Less variability, less room for improvement Opt out: Participants take action to be removed from the study. Pros: Larger, more diverse samples Cons: Less motivation and persistence, on average

158 Persistence Across Different Study Designs Example opt-in study Utility customers sign-up for a program in which they receive monthly s with tips for saving energy. Example opt-out study Utility customers receive monthly s with tips for saving energy unless they contact the utility and ask not to receive the s.

159 Designing with Persistence in Mind When measuring impact across years, the sample size estimates per group need to incorporate: After you ve implemented your study, do you plan to: Attrition rates Change the treatment? Lengthen the amount of time between treatments? Diminished effect sizes across time Stop treatments altogether?

160 Designing with Persistence in Mind Plan for long-term data collection Work with utilities to gather long-term usage data Keep detailed notes on decisions made and protocols developed Be prepared to gather additional data, like follow-up surveys

161 Example of an Effective Experiment Study Aim: Changing municipal building operators behavior regarding energy efficiency Design: 2 by 2 factorial design Use app? Yes No Frequent checkins? Yes No

162 Example: Primary Pilot Research Questions Does tracking buildings energy use lead to decreases in energy consumption? Does accountability lead to decreases in energy consumption? When used together, do these interventions lead to more behavior change than when used alone?

163 Example: Research Questions About Persistence Do the interventions lead to longterm changes in energy-related behavior? After a certain period of time, can one of the treatments be removed without leading to increases in energy use? Can both treatments be removed?

164 Example: Setting Up Tests for Persistence How this pilot planned for persistence: Plan to implement a survey to ID persistence of behaviors Developed viable hypotheses Designed the study to drop one intervention after time

165 Wrap Up