2013/4/28. Experimental Research

Transcription

1 2013/4/28 Experimental Research

2 Definitions According to Stone (Research methods in organizational behavior, 1978, pp 118), a laboratory experiment is a research method characterized by the following: the research takes place in an artificial setting, i.e. one created by the experimenter for the purpose of studying a phenomenon; the researcher assigns subjects to treatment and control conditions; the researcher manipulates one or more independent variables and assesses their impact on the dependent variables; the experimenter has control over virtually all the independent and intervening variables that affect the dependent variables.

3 Purpose of conducting laboratory experiments According to Kerlinger (Foundations of Behavioral Research, 1986), laboratory experiments serve the following purposes: they allow for the testing of predictions derived from theory or inferences drawn from other studies by providing a means for studying relationships under controlled, un-confounded conditions; and they can be use to build theoretical systems. It is easier to observe any inconsistencies with the theory in the controlled settings of laboratories.

4 Experimental Research used specifically for testing cause-effect relationship (hypotheses) Causal analysis requires: temporal priority: suspected cause precedes the effect control over variables: dependent variables: effect independent variables: causes random assignment: all participants given an equal chance to experience any given level of the independent variable

5 Alternative Explanations History Effects uncontrolled event altering participants responses Maturation Effects any process involving systematic change over time, regardless of specific events Testing Effects changes in response causes by measuring the dependent variable

6 Alternative Explanations Instrumentation Effects changes in the manner in which the dependent variable is measured Statistical Regression Effects extreme scores are likely results of random measurement error; if measures again scores tend to regress towards norm Selection Effects produced by the manner in which participants are chosen

7 Alternative Explanations Mortality Effects caused by loss of participants mid-way Participant bias participants intentional effort to alter resp. Experimenter bias experimenter s differential treatment of experimental groups

8 Impact of Alternative Explanations assumption of experimental research is that changes in the independent variable causes changes in the dependent variable hence important to rule out/account for alternative explanations to maintain INTERNAL VALIDITY

9 How to rule out alternative explanations? history effects, maturation effects, statistical regression effects, testing effects, selection effects, and mortality effects may be ruled out through random assignment of variables (i.e. conditions created by manipulating the indep. var.) instrumentation effects, participant and experimenter bias may be ruled out by controlling the experimental situation carefully

10 Summary experiments are done by: manipulating independent variable randomly assigning participants to different levels of the independent variable controlling or ruling out alternative explanations measuring responses via dependent variable experimental results can only show causeeffect relationship can occur, not that it always occur

11 Experimental Design Principles Bias Minimization E.g. common method bias, social desirability bias Proper Sampling Ideally random Be careful on generalization

12 Experimental Design Principles Randomization If a random sample can be selected, then statistics can be used to generalize, drawing conclusions about the entire population. Randomization is thus at the heart of a true experiment. Randomization also helps remove bias, where choices by the researcher, subject or others may lead to invalid results.

13 Experimental Design Principles Randomization Selection and assignment random selection random assignment Probabilistic equivalence Randomization creates probabilistic equivalence, where multiple test groups are declared statistically equivalent. Typically, a 95% equivalence (alpha <= 0.05) is sought.

14 Experimental Design Principles Noise reduction The greater the signal-to-noise, the greater the effect. There are two ways of improving the signal-to-noise ratio: increasing the signal or decreasing the noise. e.g. Factorial design experiments increase the signal, whilst covariance or blocking designs aim to decrease noise.

15 Experimental Design Principles Groups Experiments often use multiple groups, with each group forming a mini-experiment on its own, and with the findings from each group then being compared for further comparison, analysis and conclusion. Control group A control group is one which includes the same type of people (preferably randomly assigned) as those in the treatment group.

16 Types of Experiments

17 Design Notations Basic symbols Observation, O Treatment, X Assignment symbols Random assignment, R Non-equivalent groups, N Sometimes you deliberately want to work with different groups, for example men and women. Assignment by cutoff, C Sometimes assignment to groups done by a pragmatic method based on such as the sequence of arrival.

18 Design layout Sequence Design Notations Parallelism

19 True Experiments True experiments use randomized choice, selecting subjects and methods in a way that prevents bias in results - important when seeking to demonstrate a cause-and-effect relationship. In design notation, the letter R is used to show randomized assignment and a simple test with a control might look like this:

20 Quasi-experiments Quasi-experiments do not use proper random assignment, typically they recruit people in a way that can cause bias, such as using 'people on the street'

21 Non-experiments Non-experiments make no attempt to conform with experimental concerns such as randomized selection of participants or use of control groups. Note that this does not make them invalid and many useful surveys are carried out in this way.

22 Experimental Design (no. and arrangement of independent variable levels in a research project) Basic Design (Post-test-Only) subject selection random assignment indep. var indep. var. (diff. level) (diff. level) measure response measure response

23 Control-group design Two parallel experiments are set up, identical in all respects except that only one includes the treatment being explored by the experiment. The control group may have no treatment, with nothing happening to them, or they may have a neutral treatment, such as when a placebo is used in a medical pharmaceutical experiment.

24 Post-test only A common form of experiment is to apply the treatment and measure the results, for example a training course is followed by testing their knowledge as compared to a control group who are not given the training.

25 Experimental Design Basic Design (Pre-Test Design) subject selection random assignment measure dep. var measure dep. var. (pre-test) (pre-test) ind. var. (diff. level) measure dep. var. (pos-test) ind. var. (diff. level) measure dep. var. (post-test)

26 Pre-test and Post-test A problem with the post-test only is that there is no direct indication of what actual change was found in the treatment group. This is corrected by measuring them before and after the treatment. The control group is still useful as additional factors may have had an effect, particularly if the treatment occurs over a long time or in a unique context.

27 Soloman 4-group Design (can assess testing-treatment interactional effect) Subject Selection Random Assignment measure dep. var.... measure dep. var. (pre-test) (pre-test) ind. var. (diff. level) ind. var.... (diff. level) measure dep. var. (post-test) measure dep. var. (post-test) meas. dep. var. (post-test) meas. dep. var. (post-test)

28 Solomon Four-Group design Use this design when it is suspected that, in taking a test more than once, earlier tests have an effect on later tests, for example by learning or priming effects. In addition to the basic pre-test/treat/post-test design, do three additional tests, one without the treatment, one without the pre-test and one without both pre-test and treatment.

29 Solomon Four-Group design

30 Solomon Four-Group design In a test where there is no priming or learning effect, the pre-test and scores without treatment will all be similar. Where there is a priming or learning effect, then repeated tests without the treatment will show a significant change, whilst posts-tests without a pre-test will give results dissimilar to the basic pre-test and post-test design.

31 Example In a teaching experiment the Solomon design shows that testing before and without treatment have similar results, whilst results after teaching are significantly improved. This indicates that the treatment is effective and not subject to priming or learning effects.

32 Example In another experiment, the initial test seems to indicate that teaching has an effect. However, (b) shows that without teaching the score significantly improves, (c) shows that without a pre-test the score is not as impressive as (a). The single post-test (d) gives a score similar to pre-tests, as might be expected.

33 Discussion Pre-test and post-test are common ways of determining change caused by a treatment, but they are subject to improvement effects. Priming occurs when the pre-test helps the subject predict what to expect in the post-test. Learning occurs when the pre-test acts as a practice, such that the subject increases skill at doing this type of test. The Solomon design applies different variations of the test, omitting various elements and thus allowing the effects of these omissions to be assessed. Note that for a reliable result, several sets of four tests should be applied and the means used.

34 Factorial design For each variable (or factor) to be explored in an experiment, first identify the settings of each variable (or levels) that are to be tried out. To explore all combinations of factors and levels, the total number of experiments that are needed is the product of the numbers of levels.

35 Factorial design Thus with two factors F 1 and F 2, and three levels for F 1 (L 11, L 12, L 13 ) and two factors for F 2, (L 21, L 22 ), the number of experiments is 3 x 2 = 6 combinations of levels. Each experiment may be done with the same group or different groups. Factorial experiments often stick to two variables as it becomes geometrically more complex when there are more. The results may be plotted by row or column (for 2 x 2 experiments) in a graph

36 Example An investigation into the factors that cause stress in the workplace seeks to discover the effect of various combinations of three levels of background noise and two levels of interruption. They apply the test to the same group at the same time of day and day of week over six weeks. The change in stress as measured with a standard instrument

37 Example

38 Example This is shown in the graphs below (different views of the same data), from which it may be noted (amongst other things) that medium background noise has a much greater effect on stress when there are frequent interruptions as compared with when interruptions are infrequent.

39 Discussion Factors are major independent variables. Levels are subdivisions of factors. Factorial designs for the analysis of multiple variables at once, which can be very helpful when it is not sure which is more significant or how they interact. The number of experiments that are required for a full analysis increases geometrically with the number of levels. For example an experiment with four factors and three levels each would need 3 4 = 81 experiments. Factorial designs improve the 'signal-to-noise' ratio in an experiment by increasing the signal.

40 Factorial Design (test more than 1 independent variable) Subject Selection Random Assignment 1st ind. var. Random Assignment 1st ind var Random Assignment 2nd ind var 2nd ind var 2nd ind var 2nd ind var measure response measure response measure response measure response

41 Summary Basic design: participants are randomly assigned to different groups representing different levels of a manipulated independent variable Soloman 4-group test can measure testing effects, but is inefficient wrt testing the research hypothesis Factorial Design measures main effect: due to one variable interaction effect: caused by a combination of 2 or more variables

42 Quasi-Experimental

43 Quasi-Experimental Research approximates experimental design but lacks random assignment used when real experimental designs are impossible, impractical, or unethical instead of finding whether an ind. var. causes the dep. var., it tries to find out whether the ind. var. is an indicator of what the real cause is

44 Time-Series Design (dep. var. is measured repeatedly before and after the intro. of the ind. var.) Interrupted Time-Series Design ind. var. is an interruption of on-going activities O(1) O(2) O(3) O(4) O(5) O(6) O(7) O(8) O(9) ind. var. introduced O(t) ind. var. removed t

45 Multiple Time-Series Design (2 sets of observations- with and without ind. var.-are comparied) GROUP 1 O(1) O(2) O(3) O(4) O(5) O(6) O(7) ind. var. applied here to group 1 only O(t) Use correlation stats. to see if gp1 indeed differs from gp2 group 1 group 2 t

46 Longitudinal study Time studies A longitudinal study takes place over a period of time, repeatedly examining the same population or situation to determine how it changes over time. Repeated measures A repeated measures study typically makes relatively few sets of measurement. This may give reason for a relatively complex or expensive measurement that would be more problematic if repeated more often. Analyzed with a repeated measures ANOVA

47 Time series Time studies A time series study makes many measures over time, possibly in separate surveys. A trend study selects different samples over time and seeks differences to identify overall population shifts. A panel study selects a single group and gets them to comment about something at different points in time, often done as a discussion. C.f. A cross-sectional study takes place at a single point in time, typically sampling a population. The study may be complex, timebound or both. If this study proves interesting, it may be done again later, extending it into a repeated-measures longitudinal study.

48 Regression-Discontinuity Design Time-Series designs are longitudinal designs: same partcipants are repeatedly measured over time Reg.-Dis. design is cross-sectional design one measurement of different groups that represent different time periods

49 Regression-Discontinuity Design Groups 1-4 : no experience of the ind. var. Groups 5-8: different levels of exp. with the ind. var. Based on obervations on groups 1-4, use regression to project their expected dep. var. as if they have experience the ind. var. like groups 5-8 ind. O(t) Actual var. app. Projected t

50 Regression-Discontinuity design The regression-discontinuity design uses a cut-off selection method, for example where subjects are selected based on scoring above or below a certain value on a previous test. the control group is made up of those who fall at the other side of the cut-off score. The sample is thus cut in two, with one group as control and the rest as the treatment group.

51 Example A training module is designed to increase the visual-spatial ability of people with lower 'IQ'. A sample is selected and tested for IQ, with those scoring below 100 being allocated to the cut-off group. A test for visual-spatial ability was then administered before the lower-iq group was given the training. A post-test score showed that their ability had increased in this area.

52 Example

53 Discussion The name 'regression discontinuity' arises because the assignment by score causes a discontinuity across this score boundary between pre-test and post-test and between treatment and control groups, as in the diagram below.

54 Non-equivalent Groups Basis Pretest Design Use this test when it is suspected that the treatment group may be affected by factors outside the treatment topic (or that unwanted 'leakage' of the treatment is occurring). This design uses only a single group, but applies two tests to them, covering separate topics. The treatment is applied only with regard to one of these topics. Thus all people in the group take two pre-tests and two post-tests.

55 Example A History teaching method is being tested in a school with a single class. Standardized tests are used in both History and English. The score below shows that whilst the History score increases, the English score has not increased. It thus may be concluded that any change in English teaching or other factors has not affected the increase in the History score.

56 Non-equivalent Groups Basis Pretest Design Subject Selection Preexisting Differences I(1) O(1) ind. var. resulting from preexisting differences I(2) O(2)

57 Discussion In this design, the treatment group is effectively its own control group. The actual control, though, is the additional variable. Note that this 'control' variable (English, in the above example) should be similar enough to the treatment subject to be affected by the same sort of factors.

58 Discussion This design can be used in two ways. it can check for the possibility of changes elsewhere contaminating treatment results, for example where a change in teaching in another subject contributes to changes in results in the treatment subject. The design can also check for 'leakage', that the treatment is effective in one area only and not in other areas.

59 Single-Participant Design (involving only one respondent) Case Study Design intensive study of a singly participant over an extended period of time the participant is subjected to different levels of the ind. var. and the corresponding dep. var. measured good for homogenious population though insufficient for generalisation, OK for exporatory study or developing research hypothesis

60 Single-Participant Design Baseline Design Participant O(b) dep. var. baseline measure preindependent var. measures are compared with postindepend var. measured I apply ind. var. O(p) dep. var. post-test

61 Strengths and Weaknesses of Laboratory Experiments internal validity refers to the validity of a researcher's conclusion that the relationship between independent and dependent variables is causal. external validity is the degree to which the causal relationships observed can be generalized to other populations, settings, and times.

62 Strengths: High internal validity is one of the major advantages of laboratory experiments. The various threats to internal validity can be reduced or eliminated by the use of experimental design. For example, the experimenter can randomly assign subjects to experimental and control groups. The experimenter can also control the manipulation of the independent variables. Control, such as effecting truly random assignment and isolating and controlling the influence of extraneous variables not relevant to a study, is easier for the laboratory experimenter. Replication of an experiment is far easier in a laboratory than in the field. The laboratory setting allows an experimenter to create conditions or develop information systems that do not necessarily have real-life counterparts. This is especially important in the IS area for testing the feasibility of new ideas and systems.

63 Weaknesses The major criticism of laboratory experiments relates to external validity: laboratory lacks realism, i.e. there is a lack of correspondence between natural events and events in the laboratory independent variables in the laboratory are weak relative to the same variables in real-life it is not possible to manipulate more than a few independent variables in the laboratory the range of criterion situations to which a set of laboratory findings is applicable is limited.

64 External validity can be enhanced by replication both within a study and across studies. External validity can also be increased by choosing a target population to which it is to be generalized and drawing a representative sample for that target. Alternatively, several classes of persons, settings, etc. can be sampled.

65 Want to see how lab studies are done? (My research papers) "Web Strategies to Promote Internet Shopping: Is Cultural- Customization Needed?" MIS Quarterly, Vol.33, No.3, 2009, pp (with C.L. Xia, K.H.Lim, K.Leung, W.Huang, I.Benbasat) How Do I Trust You Online, and If So, Will I Buy?: An Empirical Study of Two Trust Building Strategies, Journal of Management Information Systems, Vol.23, No. 2, Fall 2006, pp (with K.H.Lam, C.L.Sia, I.Benbasat)

66 Drawing Conclusions (Conclusion Validity) 66

67 Inferring Cause Two variables change at the same time is not proof of cause Hume's temporal regularity three basic conditions that are necessary for cause and effect to be inferred: Cause and effect must occur close together in time The cause must occur before the effect The effect should never occur without the cause occurring first

68 Hume's temporal regularity Some Problems: There may be a significant delay between cause and effect. For example a person may be bitten by a mosquito and die some time later. The cause and effect may occur so closely together in time it is impossible to measure the time difference between them. Effects can be caused by multiple things. For example a person can die of things other than an insect bite. A key issue here is that there can be multiple causes which have to occur in sequences or together for effects to happen. Another problem is that just because A follows B it need not happen next time.

69 Mill's Formulation of Hume s Rules Cause must precede effect Cause and effect must correlate (when one changes, the other also changes in a proportionate way) All other explanations of the cause-effect relationship must be eliminated

70 Mill's induction Mill s three methods of inferring cause: The method of agreement: the effect is present when the cause is present The method of difference: the effect is absent when the cause is absent The method of concomitant variation: When 1 and 2 are demonstrated, the case for causal connection is made stronger by eliminating other possible causes Calls for controlled experiments!

71 Cook and Campbell's three criteria (1979) Covariation: Changes in the assumed cause (X) are related to changes in the assumed effect (Y). Changing X results in a predictable change in Y. Temporal Precedence: The assumed cause must occur before the asssumed effect. No Plausible Alternative Explanations: The assumed cause must be the only reasonable explanation for changes in the measured assumed effect.

72 Experimental structure (to infer cause-effect) Vary the likely cause as the independent variable and measure the likely effect as the dependent variable Monitor other factors that may influence the situation, particularly those that might have some causal effect. Control other factors as far as possible and monitor those that cannot be held steady.

73 Experimental structure (to infer cause-effect) Have a separate 'control' experiment in which the cause is not present, but all other factors remain the same. Repeating the experiment a number of times to ensure the results are not random and that probabilistic causality can be assessed Varying other factors across multiple experiments to determine whether the presence or absence of these factors is significant

74 Two error types 74

75 Type 1 Error The Type 1 error (often written 'Type I error') occurs when it is concluded that the primary hypothesis, H 1 is true, whilst it is actually false. In other words the experiment falsely appears to be 'successful' in proving the primary hypothesis. The probability of making a Type 1 error is often known as 'alpha For statistical significance to be claimed, this often has to be less than 5%, or 0.05

76 Type 2 Error The Type 2 error (often written 'Type II error') occurs when it is concluded that the primary hypothesis, H 1, is false, whilst it is actually correct. The probability of making a Type 2 error is known as 'beta' Cohen (1992) suggests that a maximum acceptable probability of a Type 2 error should be 0.2 (20%)

77 Type I and Type II Errors Accept P1 Reject P1 P1 is true Correctno error Type II error P1 is false Type I error Correctno error 2013/4/28