Conducting Experiments, Survey Construction and Data Collection RCS 6740 6/28/04 Practical Considerations of Research This lecture will focus on some of the practical aspects of conducting research studies such as: How do you select research participants What should you consider when deciding how to manipulate an independent variable How do you measure a variable What should you consider when constructing a survey What should you consider when collecting data How do ensure control in your study Selecting Research Participants Define your participants People with Aphasia Women with Breast Cancer Rats with 5 Legs Define your population Everyone in the United States People in RCS 6740 Consumers at The Center (CIL)
Selecting Research Participants Define your sampling technique Probability sampling Random sampling Stratified random sampling Systematic sampling Cluster sampling Non-probability sampling Quota sampling Determine your sample size Remember, larger sample sizes may result in more significant data Manipulating the Independent Variable Construct the Operational Definition of your variables Exercise as defined by jogging for 20 minutes, 3 times a week, for 6 weeks Depression as defined by the Geriatric Depression Inventory Set the Stage Provide the participants with Informed Consent Usually a document that entails the name of the study, purpose(s) of the study, potential benefits and risks, funding sources, contact information, and a statement that they can withdraw at any time Explain to the participants in more detail why the study is being conducted Rarely are hypotheses given to participants Tell them the study looks at employment when the hypothesis looks specifically at bias towards the employment of people with disabilities Can be completely truthful and precise Deception may be used When deception is used, it is imperative that participants are told the truth during a debriefing at the conclusion of the study Types of Manipulations Straightforward Manipulations Presenting written, verbal, or visual material to participants Examples: Having participants read material on substance abuse prevention Read a passage to a participant and ask them questions later to test recall (memory) Have a computer program that presents different words that someone needs to pronounce to measure the level of aphasia Most research is done in a straightforward manner Researchers can vary the difficulty of material, motivational levels, the way questions are asked, or the characteristics of the participants
Types of Manipulations Staged Manipulations Also called Event Manipulations Used for 2 reasons The researcher may need to create some sort of psychological state (anxiety) The researcher may need to stage a manipulation to recreate a real-world scenario Having a participant do one task and then having them do more tasks at the same time Staged manipulations usually employ a confederate Also called an accomplice A confederate is someone who appears to be another particpant in an experiment but is really a part of the experiment Example: Someone who purposely insults a participant in a study in order to provoke anger or frustration Types of Manipulations Another example of the use of confederates: Asch (1956) study on conformity Which line is bigger? 1)--------------------- 2)----------------------------- 3)------------------- Right before a participant had to choose which line was the longest, a confederate announced an incorrect answer Repeatedly, Asch found that people conformed to the confederate s incorrect response Types of Manipulations Straightforward Manipulations Vs. Staged Manipulations: It is easier to recreate real-world situations using Staged manipulations Staged manipulations are more extensive in terms of cost and set up procedures It is difficult for other researchers to replicate staged manipulations Data and results of Straightforward manipulations are easier to interpret due to a greater sense of control
Strength of Manipulations The strength of a manipulation can effect how results are interpreted Example: In a study on the use of a new medication to treat diabetes, it is important that the researcher uses a sufficient dosage of the drug in order to see an effect 5 mgs of Diabetex does not reduce blood sugar levels 10 mgs of Diabetex reduces blood sugar levels by 3% 20 mgs of Diabetex reduces blood sugar levels by 10% Strength of Manipulations A strong manipulation maximizes the differences between groups and increases the chance that the independent variable will have a significant effect on the dependent variable Strong manipulations are especially important in the early stages of research (Why?) Although you want a manipulation to be as strong as possible, ethical considerations should be examined You do not want to make someone so anxious that they decide to jump off a building Cost of Manipulations Most researchers are limited in terms of the funding they have for a research project Confederates and elaborate manipulations may cost a great deal The straightforward approach is usually more budget friendly
Measuring the Dependent Variable Types of Measures Self-report measures Behavioral Measures Physiological Measures Measuring the Dependent Variable Self-report Measures Used to measure attitudes, judgments about oneself or another, emotional states, confidence in oneself or others, and other aspects of human thought and behavior Example: Please use the following 7 point scale to answer this question. I feel that I am the best student in this class: Strongly Disagree _ Strongly Agree Measuring the Dependent Variable Behavioral Measures Direct observations of behaviors such as: How many errors someone makes on a test How many times a person can lift a 10lb object How may free throws a person makes in 30 minutes Things to consider with behavioral measures Rate: How many times a behavior occurs Reaction Time: How quickly a response occurs after a stimulus Duration: How long a behavior lasts
Measuring the Dependent Variable Physiological Measures Recordings of responses of the body Galvanic Skin Response (GSR): Measures the electrical conductance of the skin to look for arousal or anxiety Electromyogram (EMG): Measures muscle tension to look for stress Electroencephalogram (EEG): Measures activity of brain cells such as looking at what area of the brain is active while dreaming Magnetic Resonance Imaging (MRI): A technique used to gather detailed images of the inner body Functional Magnetic Resonance Imaging (fmri): Looking at the inner body while someone performs a physical or a cognitive task Measuring the Dependent Variable What Type of Measure(s) to use? Basically, what type(s) of measure a researcher will use depends on the nature of the study, cost, and possible ethical considerations If at all possible a researcher should use a combination of all three types: To study anxiety, a researcher can administer a selfreport measure asking how nervous a person thinks they are, a behavioral measure that looks at how many times a person s voice cracks, and a physiological measure looking at fluctuations in someone s heart rate When administering multiple measures, it is important to remember to account for a possible order effect Example: A self-report measure on how nervous a person is might cause them to become more nervous thus increasing their heart rate Measuring the Dependent Variable Sensitivity of the Dependent Variable The dependent variable should be sensitive enough to detect differences Do you like your boss (not sensitive) On a scale of 1 to 10, how much do you like your boss (very sensitive) Beware of the Ceiling Effect and the Floor Effect Ceiling Effect: Task is too easy so everyone performs well Lifting a 1lb object Floor Effect: Task is too difficult so everyone performs poorly Lifting a 200lb object
Information on Survey Construction DESIGNING SURVEYS A good question is one that produces answers that are reliable and valid measures of something that we want to describe 2 types of question evaluations: Those aimed at evaluating reliability Those aimed at assessing the validity of answers Survey Construction 5 process standards of questions and answers Questions need to be consistently understood. Questions need to be consistently administered or communicated to respondents. What constitutes an adequate answer should be consistently communicated. Unless measuring knowledge is the goal of the question, all respondents should have access to the information needed to answer the question accurately. Respondents must be willing to provide the answers called for in the question. Survey Construction Possible steps to assess the extent to which questions meet process standards Focus group discussions Intensive or cognitive interviews Field pre-tests under realistic conditions
Survey Construction Some general rules for designing good survey instruments The strength of survey research is asking people about their firsthand experiences: what they have done, their current situations, their feelings and perceptions. Questions should be asked one at a time. A survey question should be worded so that all respondents are answering the same question. If a survey is to be interviewer administered, wording of the questions must constitute a complete and adequate script such that when the interviewer reads the question as worded, the respondent will be fairly prepared to answer the question. All respondents should understand the kind of answer that constitutes an adequate answer to a question. Survey instruments should be designed so that the tasks of reading questions, following instructions, and recording answers are as easy as possible for interviewers and respondents. Survey Construction Ways of addressing validity in surveys Deriving questions from relevant literature Expert panel Data Collection Tips Use more than one survey or measure Come prepared Extra measures, pens, equipment Be prepared to answer questions about the survey Important not to sway a response Follow up to ensure all items were filled out (999=missing, 888=don t know) Know your survey How to score it, the factors, the subscales If possible, have a limited amount of researchers collect the data Enter data as quickly as possible and use a trusting computer program (SPSS, ECXEL) Have two people enter the data (one reads, one enters)
Control Considerations Participant Expectations When a participant knows too much about a study, they may expect to perform in a certain manner Demand Characteristics: any feature of an experiment that might inform participants of the true purpose of a study Once someone is aware of the hypothesis of a study, the usually do whatever it takes to confirm the hypothesis Hypothesis: People who take a drug will lessen their anxiety levels. Once a participant knows this, they may choose to do whatever they can do become less anxious (e.g., exercise, see a counselor, meditate). Control Considerations Ways to Control Participant Expectations Use deception to cover up what is really being studied Administer irrelevant measures Add filler questions to measures Filler questions are unrelated questions on a survey that should disguise the true purpose of a study Include a Placebo group in a study Placebo effect: When it is not know if results of a treatment can be attributed to the treatment itself or the expectancies of the treatment Placebos may consist of giving someone a pill that is made of sugar or injecting someone with saline solution It is important to remember ethical consideration of using a placebo group. For example, if someone in the placebo group becomes very ill, you may want to administer treatment to them Control Considerations Experimenter Expectations Sometimes experimenters who know the purpose of a study may expect a participant to act or perform a certain way Also known as expectancy bias There are 2 potential sources of experimenter bias When a researcher treats participants in the control group differently than participants in the treatment group Spending more time with the treatment group Researchers may record or analyze observations in a biased manner Well, even though he spoke 10 words clearly, I thought he was close on another so I will give him 11/20
Control Considerations Ways to Control Experimenter Expectations Researchers should be well trained and highly ethical Run all conditions simultaneously so the experimenter s behavior is the same for all participants Use experimenters who are unaware of the true hypothesis of the study Double-Blind Study: Neither the participant nor the experimenter knows whether a placebo or the actual treatment is given Once again, it is highly unethical for experimenters to influence participants or manipulate data!!! Additional Tips for Conducting Experiments Once a study is designed, there are certain ways to improve it Write up a thorough Research Proposal Includes a literature review, purpose of the study, procedures of the study, copies of measures, a budget, and the manner with which the data will be analyzed Used often for grants, Institutional Review Boards, and Ethics Committees Conduct a Pilot Study A trial run of a study with small sample sizes Used to get the kinks out Used to gather participant feedback Additional Considerations of Experiments Manipulation Checks A manipulation check is a way to measure if the independent variable has an effect on the dependent variable Checks if the independent variable manipulation is a successful operationalization of the conceptual variable being studied Provides evidence for Construct Validity Allows researchers to change something in a study before it is too late Allows researchers to see if they are getting significant results Example: In a study looking at whether people are nicer to attractive people, a researcher will determine whether the person who they chose to be attractive really is being viewed as so by the participants. If they are not viewed as attractive, the researchers can hire another model. Manipulation Checks may consist of any type of measure (i.e., selfreport, behavioral, physiological)
Additional Considerations of Experiments Debriefing Debriefing is the act of meeting with the participants after the conclusion of the study to discuss the true purpose of the study, ethical considerations of the study, and educational implications of the study Can also be used to ensure that the participants do not share any information with those still involved in the study Can also be beneficial as researchers may gain valuable information about how the study looks and feels from the inside Analyzing Data After a quantitative study has concluded, all data should be analyzed statistically Communicating Research to Others Once a researcher has finished a study, they should communicate their results and findings to others Professional Meetings and Conferences (NRA) that present the opportunity to share results in a verbal, written, or poster format Journal Articles also are a good way to share results Peer Review is the process of a paper being accepted or declined entry into a journal based on reviews from peers About 90% of papers submitted to the more prestigious journals are rejected but do not let this get you down