MBACATÓLICA JAN/APRIL 2006 Marketing Research Fernando S. Machado Week 5 Experimentation 1 Experimentation Experimental Research and Causality Experimental Designs Lab versus Field Experiments Test Marketing 2 1
Experimental Research Experiment: Manipulation of a treatment (independent) variable (x), followed by observation of response (dependent) variable (y) Test units: individuals, organizations or other entities, whose response to treatments is being studied. Experiment must be designed to control for other (extraneous) variables to establish causal relationship 3 What Constitutes Causality? A change in variable will produce a change in another variable The occurrence of X increases the probability of the occurrence of Y Conditions for valid Causal Inference Condition of concomitant variation: evidence that a strong association exists between an action and an observed outcome Condition of time order of occurrence: evidence that the action preceded the outcome Absence of competing causal explanations: evidence that there is no strong competing explanation for the relationship that a high level of internal validity exists 4 2
A Classification of Experimental Designs Experimental Designs Pre-experimental True Experimental Quasi Experimental Statistical One Group After-Only Two Group After-Only Time Series Randomized Blocks One Group Before-After Two Group Before-After Multiple Time Series Latin Square Non-matched Control Group Solomon Four- Group Factorial Design 5 Basic Symbols and Notations O denotes a formal observation or measurement. X denotes exposure of test units participating in the study to the experimental manipulation of treatment. EG denotes an experimental group of test units that are exposed to the experimental treatment. CG denotes a control group of test units participating in the experiment but not exposed to the experimental treatment. R M denotes random assignment of test units and experimental treatments to groups. denotes that both the experimental group and the control group are matched on the basis of some relevant characteristics. 6 3
Pre-experimental experimental Designs One Group, After-only Design (One shot case study) Apply the experimental treatment to a subject or group and measure the results EG: X O Problems: Leaves open the possibility that the results could be explained by events external to the design (history effects) Maturation and selection effects may also be present It does not provide the basis of comparing the level of O to what would happen when X was absent. 7 Extraneous variables History specific events that are external to the experiment but occur at the same time as the experiment. Maturation extraneous variable attributable to changes in the test units themselves that occur with the passage of time. Mortality extraneous variable attributable to the loss of test units while the experiment is in progress. ME normally is not uniform across the sample. 8 4
Extraneous variables Main testing effect effect of testing that occurs when a prior observation affects a later observation. Interactive testing effect effect in which a prior measurement affects the test unit s response to the independent variable. Selection extraneous variable attributable to the improper assignement of test units to treatment conditions. Instrumentation extraneous variable involving changes in the measuring instrument or in the observers themselves. 9 Pre-experimental experimental Designs (Contd.) One-group, Before - After Design (One group pretest-posttest) Improve control by adding before measure EG: O 1 X O 2 Before measure adds sensitivity by adding another method to control for confounding variables Treatment effect is measured as TE=O 2 -O 1 10 5
Pre-experimental experimental Designs (Contd.) Threats to Experiment Validity: History, Maturation and Selection Effects Before Measure (Testing) Effect May alert respondents to the fact that they are being studied, causing: Main testing effect (O1 O2) Interactive testing effect (O1 X) Mortality Effect Some subjects may stop participating in the experiment Instrumentation Effect Results from a change in the measuring instrument 11 Pre-experimental experimental Designs (Contd.) Nonmatched Control Group Design (static group design) Introduce a control group to control for history and maturation EG: X O 1 - - - - - - - - - CG: O 2 History, maturation, testing (main and interactive) and instrumentation effects have been eliminated. Problems Potential selection bias remains (and possibly also mortality bias) 12 6
Pre-experimental experimental Designs (Contd.) Matched Control Group Design Matches experimental and control groups to reduce selection bias EG: M X O 1 - - - - - - - - - - - CG: M O 2 Problem Test units can be matched on only a few characteristics, so they may be similar on the variables selected but unequal on others. 13 True-experimental experimental Designs True experimental designs adopt random assignment procedure and use one or more control groups Random Assignment For any given assignment to a treatment, every member of the universe has an equal probability of being chosen for that assignment 14 7
True-experimental experimental Designs (Contd.) Two Group, After-only Design Randomization can match test and control groups on all dimensions simultaneously, given a sufficient sample size EG: R X O 1 - - - - - - - - - - - - - - CG: R O 2 There is no interaction effect of testing as there are no pretest requirements. Problems Elimination of selection bias can not be tested for, as there is no pre-treatment measurement Potential mortality effects 15 True-experimental experimental Designs (Contd.) Two-group, Before-after Design Adds a control group to one-group, before - after design Helps control for history and maturation Controls for reactive effect of O 1 on O 2 (main test effect) EG: R O 1 X O 2 - - - - - - - - - - - - - - CG: R O 3 O 4 Problems Does not control for interactive test effect, as prior measurement O 1 may affect the reaction of the EG to the treatment. Potential mortality effect remains 16 8
True-experimental experimental Designs (Contd.) Solomon Four - Group Design EG: R O 1 X O 2 - - - - - - - - - - - - - - - - - CG: R O 3 O 4 EG: R X O 5 - - - - - - - - - - - - - - - - - CG: R O 6 Provides power to control for before measure effect of O 1 on both X and O 2 This design is often prohibitively expensive 17 Quasi-experimental Designs Provide more measurements and more information than pre-experimental design Offer some degree of control but they are not true experimental designs due to: there is no random assignment of subjects (the researcher is unable to expose test units to the treatment randomly). the researcher may lack control over the schedule of the treatments. the researcher may lack control over when and whether the test units are exposed to the treatment. 18 9
Quasi-experimental Designs (Contd.) Time Series Designs Series of measurements are employed during which an experimental treatment occurs EG: O 1 O 2 O 3 O 4 X O 5 O 6 O 7 O 8 Trend Studies Measures over time come from succession of separate random samples from the same population Continuous Panel Studies Collect a series of measurements on the same sample of test units over an extended period of time 19 Classical Types of Experimental Designs Considers only one treatment level of an independent variable at a time Statistical Allows for examining the impact of different treatment levels of an independent variable (eg. Low, medium and high price levels). Allows for examining the impact of two or more independent variables (eg. price and advertising) Specific extraneous variables can be statistically controlled (for example by using ANCOVA) In general SD are after-only designs 20 10
Statistical Designs (contd) Completely Randomized Design Any number of treatments can be assigned to test units on a random basis Example: A cultural centre wants to test 3 different promotion pieces for a series of plays. The question is which will deliver the most orders. A mailing list with 20 thousand people is available. A random sample of 1 200 is selected and divided randomly into three groups of 400. EG 1 R X 1 O 1 - - - - - - - - - - - - - - - - EG 2 R X 2 O 2 - - - - - - - - - - - - - - - - EG 3 R X 3 O 3 21 Randomized Block Design Employs the randomization process for all variables Matching ensures that there are no differences between test samples on matched variables Matching and randomization are combined Ex: Management believes that urban respondents react diferently from from suburban respondents to promotion for subscriptions to a series of plays. EG1: R X 1 O 1 Urban EG2: R X 2 O 2 EG3: R X 3 O 3 EG1: R X 1 O 1 Suburban EG2: R X 2 O 2 EG3: R X 3 O 3 22 11
% who ordered tickets Treatment Urban Suburb. Average A 11 4 7.5 B 25 11 18 C 24.5 15.5 20 Averages 20 10 15 Can we conclude that promotion C is more effective than promotion B (see hypothesis testing)? Is it possible that most effective promotion for urban area is different from most effective promotion for suburban areas (interactions)? 23 Statistical Designs (Contd.) Latin Square Design A project s research goal was to determine the effect of variations on nutritional information on canned peas labels on shopper perceptions and preferences. Four levels of information were tested: Simple quality statement (I). List of major nutrient components and indication of whether the product is high or low on them (II). List and exact amount of major nutrient component (III) List and exact amount of all nutrient components (IV). There are two control or block variables: the store and the brand. Four brands ofcanned peas (eachwitha associated price) and four different supermarkets were selected. A sample of 50 shoppers was interviewd in each supermarket. 24 12
Statistical Designs (Contd.) Stores 1 2 3 4 Private Brand A 21 cents III IV I II Private Brand B 22 cents II III IV I Major Brand A 25 cents I II III IV Major Brand B 26 cents IV I II III In a Randomized Block Design each cell would have 4 groups, one for each treatment level. Latin Square Design is more economical. 25 Results Average levels of preference levels are higher for treatment IV, but: The relationship between nutritional information and preference may not be statistically significant. This relationship is not monotonic. This design does not allow for analysis of interaction effects between treatment variable and control variables. 26 13
Statistical Designs (Contd.) Latin Square Design Reduces number of groups involved when interaction between the treatment levels and control variables are unimportant Requires same number of rows, columns, and treatment levels Cannot be used to determine interaction effects 27 Factorial Designs Statistical Designs (Contd.) Two or more experimental variables are considered simultaneously Each combination of the experimental treatment levels applies to randomly selected groups EG 1 X 1 (A1, P1) O 1 EG 2 X 2 (A1, P2) O 2. EG n X n (Ap, Pm) O n (p*m=n) Provides the ability to determine interactive effects of pairs of experimental variables and the main effect 28 14
Further Issues in Experimental Research What are the internal and external threats to the validity of the experiment? Internal validity: The extent to which competing explanations for the experimental results observed can be ruled-out. External validity: The extent to which causal relationships measured in an experiment can be generalized to outside people, settings, and time. Should the experiment be performed in a "laboratory" setting or in the "field"? 29 + - Lab Experiments Experimental treatment introduced in an artificial or laboratory setting Least costly and allow greater control over the experiment Alternative explanations of results are reduced, increasing internal validity Artificiality of the environment may cause reactive error Results may not have external validity Field Experiments Experimental treatment introduced in a natural setting (actual market conditions) Response tends to be natural Tend to have much greater external validity Difficult to control Competing explanations for results exist 30 15
Application of Causal Research: Test Marketing Test Marketing: An application of a controlled experiment done in limited, but carefully selected, parts of the marketplace, called test markets. In test marketing, the marketing mix variables (independent variables) are varied and the sales (dependent variable) are monitored so that an appropriate national marketing strategy can be identified. The two major objectives of test marketing are: To determine market acceptance of new products. To test alternative levels of marketing mix variables (retail prices, shelf space and display, etc.) 31 Simulated Test /Pretest Markets Use of survey data and mathematical models to simulate test market results ASSESSOR 1. Premeasurement for established brands (SAQ): composition of relevant set of established brands, attribute weights and ratings, preferences. 2. Respondents are exposed to advertisement of the new brand. 3. Respondents proceed to mock store setup. They are given a small amount of money and may purchase whatever brands they wish. 4. Respondents who bought the new product are called after 2-3 weeks and measured on postusage variables (new brand usage rate, satisfaction ratings and repeat purchase propensity; attribute ratings and preferences for relevant set of established brands plus new brand.) BASES 1. Shopping mall intercept interviews are done at 4 or more geographically dispersed cities (respondents not screened for category use) 2. Respondents are exposed to a concept and asked a standard set of questions, e.g. like/dislike, perceived value for money, purchase intention. 3. Those expressing a positive purchase intention are given some product to use at home. 4. After several weeks, users are called on the telephone to obtain after-use measures similar to before-use measures obtained in stage 2. 32 16
Cost Involved Limitations of Experiments Time Considerations Security Field experiment exposes marketing program in the marketplace Difficult to hide from competitors Implementation Problems Difficult to gain cooperation within the organization Contamination may occur in experiments involving market areas due to inability to confine the treatment to designated experimental area 33 17