Experimental Research I Day 3 Quiz/Review Quiz Review Normal Curve z scores & T scores More on the normal curve and variability... Theoretical perfect curve. Never happens in actual research Mean, median, mode are equal 50% of scores lie above mean, 50% lie below 68% of scores are between one SD above the mean and one SD below the mean 95% of the scores are within two SD s above and below the mean 99.7% of the scores are within three SD s above and below the mean 1
Normal Curve/Distribution Standard Scores and Their Relationships to the Normal Curve Raw score vs. Standard Score As is vs. computed to reflect the scores place with respect to the sample mean and standard deviation Standard Scores allow comparisons on different types of measures. Casts scores in same terms for easier comparison z scores - Allow raw scores on various measures to be compared z = (Raw - Mean)/Standard Deviation Expressed in SD units On data set, convert 2 scores to z scores Test 0-100 possible (M = 70, SD = 10) Calculate for raw scores of 75 & 40 T Scores A way to avoid negative scores and therefore increase interpretability T = (z X 10) + 50 T has mean of 50 & SD of 10 Convert same 2 scores from data set to T scores SAT/GRE scores SAT/GRE = (z X 100) + 500 CAVEAT - Standard scores are most beneficial when distribution resembles normal curve 2
Experimental Research Experimental Research Only type of research with an intervention A direct attempt to influence a particular variable Only method that can truly begin to untangle cause and effect hypotheses Directional Hypothesis = Theory statement predicting the outcome [directional] (There will be a significant difference ). Reflects researcher s expectations. Kindergarten students taught through the Gordon method (treatment group) will score significantly higher on a measure of pitch matching than will students taught through Kodaly or no specific method (control groups). Null Hypothesis Null Hypothesis = Theory statement predicting the outcome stated in the negative [nondirectional] (There will be no significant difference ) The statistical hypothesis that states that there are no differences between observed and expected data. Does not reflect researcher s expectations (value free) There will be no significant difference b/w the pitch matching ability of kindergarten students taught through the Gordon method (treatment group) compared to students taught through Kodaly or no specific method (control groups). The goal is to REJECT the Null Hypothesis [Form a hypothesis, then state in form of null hypoth.] 3
Hypothesis Testing Statistical tests used to make a decision about whether the data contradicts the null hypothesis. Goal in successful experiment is to reject null hypothesis A null hypothesis is never proven, as the absence of evidence against the null hypothesis does not establish it. In other words, one may either reject, or not reject the null hypothesis; one cannot accept it. Failing to reject it gives no strong reason to change decisions predicated on its truth (No reason to stop teaching Gordon just b/c experiment didn t work), but it allows for the possibility of obtaining further data and then re-examining the same hypothesis. Eventual consensus? Type I and Type II Error Type I Error is erroneously claiming statistical significance or rejecting the null hypothesis when in fact, it s true (claiming success when experiment failed to produce results) Possible w. incorrect statistical test Type II Error is when a researcher fails to reject the null hypothesis when it is in fact false (claiming failure when successful) The smaller the sample size, the more difficult it is to detect statistical significance In this case, a researcher could be missing an important finding because of study design Group Comparisons Experimental Group Receives a particular treatment specified by the researcher Control/Comparison Group Does not receive that particular treatment Sometimes difficult in educational research to have a strict no-treatment, control Example: Any instruction is likely to be more effective than no instruction Compared through statistics 4
Randomization Random assignment to groups Every individual has an equal chance of being in the experimental or control/comparison group Supposed to help eliminate extraneous sources of variance For example if the groups are sufficiently large, any differences between groups on extraneous variables are likely to be due to chance or randomly distributed among the groups Quasi-Experimental=non-randomized groups Most ed. research Intact classes & convenience samples Impacts ability to generalize to whole population Variables Independent variable (IV) The experimental or treatment variable This variable is manipulated by the researcher Examples: instructional approach, environmental condition, the introduction of a particular musical element Dependent variable (DV) Compared b/w groups The criterion or outcome variable Examples: student attitudes, student achievement, teacher effectiveness as measure by? Experiments can be expressed as The effect of the IV on the DV Extraneous Variables Those that are not specifically included in the study but never the less may effect the outcome Object is to control for extraneous variables The researcher may not know them all Manipulating the IV Presence of the variable vs. absence of the variable The use of practice charts (treatment group) vs. no practice charts (control group) One form of the variable vs. another modeling vs. verbal music instruction (vs. control group) Varying degrees of the same variable 100% positive feedback, no negative feedback vs. 50% positive feedback, no negative feedback 5
Controlling for Extraneous Variables Best case scenario all individuals are as similar as possible on all variables other than the independent variable Methods to control: Randomization & large sample Holding variables constant (freeze private lessons) Build variable into the design (compare private lessons w/ no private lessons) Matching pairs one to control, other to exper. Within-subject design-all groups receive treatment(s) and no treatment Statistical control analysis of covariance (ANCOVA) Design and Experiment [Effect of Colored note heads on Music Reading] State Hypothesis and Null Hypothesis Select sample and assign to group (control and treatment). How many in each? Identify independent and dependent variables. Any possible extraneous variables? Describe experiment. What will you do w/ each group and for how long? How will you know what they already know? Nomenclature/Abbreviations When looking at the symbols used to describe various experimental design approaches: R = random assignment O = testing (pre- or post-) X = treatment C = control/comparison M = matched 6
Pre-Experimental Designs [Pilot Studies Generally Weak] One Shot Case Study (X O) No random assignment, No control/comparison, no pretest One-Group Pre-test, Post-test (O X O) No random assignment, No control/comparison group Static/Intact-group Comparison X O No random assignment O Static/Intact-group Pre-test, Post-test No random assignment, possible pre-test effects O X O O True Experimental Designs Stronger not always possible in educ. Randomized Post-test Only, R X O Control Group R O Still not sure about pre-test levels Randomized Pre-Test, R O X O Post-test, Control Group R O O Good checking whether smaller groups are actual similar at the start of the study and possible effects of pretest Randomized Solomon Four-Group & Posttest Only, Control Group Solomon 4 Group controls for possible sensitization effects due to testing or maturation. R O X O R O O (maturation or testing?) R X O (effect of pretest?) R O (control group) Posttest only, control group R X O R O No baseline established. Control group might have improved as much but started at a lower level 7
Quasi-Experimental Design So called b/c there is no randomization Matching Only Participants matched in pairs to control for an extraneous variable rather than randomly assigned Counterbalanced Design (next slide) Multiple groups receive all treatment types in different order Average post-test scores across groups are compared to determine effectiveness/effect of the treatment order Vulnerable to multiple-treatment interference Time-series Design Outcome measured several times before and after introduction of the treatment O O O O X O O O O Equivalent Materials Design [O Xma O] [O Xmb O] ma = teaching method a mb = teaching method b (sight singing example) Counterbalanced Design (Latin Square) Quasi-Experimental Design Factorial Design (2 or more factors (IVs) at different levels) Examining the effect of more than one independent variable Allows for examination of moderator variables (i.e. gender, age) and interaction effects b/w combinations of IVs and the DV Example: CAI vs. traditional among high achieving and low achieving students Possible outcome showing interaction of two IV s High achieving students may do equally well w/ CAI and traditional methods, while low achieving students may do better w/ CAI vs. traditional methods. What if you had not separated these groups out? 8
Factorial Example (Two Way - 2x2) IVs = achievement level & instructional mode (trad. vs. CAI) DV = musical achievement test? Groups High achieving & CAI High achieving & traditional Low achieving & CAI Low achieving & traditional Factorial Example (Two Way - 3x3) IVs grade level beginning strings (4, 5, 6) & # of class times per week (2, 3, 4) Dependent Variable = musical achievement end of first year (Measured by a solo performance on selected piece, graded w/ rubric) Also possible to have three way (p. 168) Internal Validity - Time Bound Factors What happens within the experiment History What happens b/w pretest and posttest (private lessons, change in practice routine) Maturation is change result of treatment natural result of repetition and improvement over time?) Mortality Loss of participants may cause imbalance b/w groups 9
Internal Validity Sampling & Measurement Factors Testing pretest affect posttest. Ceiling and floor effects (eliminate outliers?) Instrumentation changes in measurement or observers (judges at contest from one site to the next) Statistical regression students who score extremely high (ceiling) or low (floor) on pretest may regress to the mean on posttest Selection participants do not represent normal population (also affects external validity) Interactions influence of a combination of the above factors External Validity Generalizability Population Validity Extent sample is representative of the population to which the researcher wishes to generalize the results. Ecological Study conditions and setting are representative of the setting in which the researcher would like to apply the findings Replication Results can be reproduced (problem w/ Mozart effect) Detailed description of the sample needed in study Important regardless of sampling method Next best thing if not a large, random sample often the case in music ed. research Consider demographic questions in descriptive research Threats to External Validity Effect or interaction of testing (testing will not occur in natural setting) Sample does not reflect population Discuss in research report Reactive effects of sample Hawthorne Effect Effects due simply to subjects knowledge of being in a study John Henry Effect Control group performs beyond usual level because they perceive they are in competition with the experimental group Teacher or Researcher interactions different than in population Subconsciously encouraging or discouraging a group Research setting does not reflect typical settings (ecological validity) A university lab school 10
For Friday Bring Laptop if you have one Rewrite abstracts and send via email Read 1 more article for your study and write an abstract Submit first draft of Introduction to your study (by 7pm Friday) Part II Lit review due Monday 11