Introduction to statistics Dr Alvin Vista, ACER Bangkok, 14-18, Sept. 2015

Transcription

1 Analysing and Understanding Learning Assessment for Evidence-based Policy Making Introduction to statistics Dr Alvin Vista, ACER Bangkok, 14-18, Sept Australian Council for Educational Research

2 Structure of workshop Lecture/presentation focus on concepts, brief review Practical exercises using the most common and accessible software Excel; Hands-on, to maximise transfer of knowledge and develop skill Discussion and interpretation of sample studies Collaborative setting If you don t know how to do something, seek help. If you know how to do something, provide help. If you re not sure, interact.

3 Drawing knowledge from reality Measurement Statistical Theory Data Theory Statistics allows us to draw knowledge or conclusions from the data. Measurement theory allows us to draw meaningful data from reality.

4 What is Measurement? A formal definition: Measurement may be regarded as the construction of homomorphisms (scales) from empirical relational structures of interest into numerical relational structures that are useful. (Krantz et al., 1971, p.9) In other words: Measurement is a process where a variable (or construct) can be converted into a number in a consistent manner.

5 Data and Models DATA Observations Measurements Sensory perceptions MODELS Theories Interpretations Generalisations

6 Data and Models DATA What you see (observed) MODELS What Google Maps says

7 If there is a mismatch between data and model, which is more likely to be wrong? Data and Models Data cannot be changed but the methods to collect data can be improved to increase the quality of data.

8 Drawing knowledge from reality We can do statistical analysis ONLY AFTER we re confident that our data is reliable Measurement Statistical Theory Data Theory Better data = better fitting models Better fitting models = better understanding of reality

9 Fundamentals: Statistics Statistics is the study of data. It concerns with the: Collection Analysis Presentation Interpretation of data.

10 Data in education research In educational context records are usually students or schools or parents variables are usually responses of the students to the test items or responses of students or school principals or parents to the questionnaire items

11 Data: values for variables & records In the educational data responses to a particular item from all respondents form the values for the corresponding variable a column of values in our imaginable table responses from a particular respondent to all items form the values for the corresponding record a row of values in our imaginable table

12 Levels of measurement Nominal: Denote a category; statistics include counts such as mode and frequency distributions Ordinal: Rank order is described but successive categories do not denote equal differences of the measured attribute; statistics include median Interval: Where the measurement is presumed to denote equal intervals between scores. Both the base point and unit of measurement are arbitrary Ratio: Note that ratio scales have a natural base value that cannot be changed (i.e., a zero in one unit means the same in all other units). Only the unit of measurement is arbitrary. Non-metric -- categorical measures which describe differences in type or kind; arithmetical operations are not applicable Metric -- continuous measures which reflect differences in amount or degree

13 In a nutshell Level of measurement has direct implications for how relationships within and between variables can be contained and identified

14 Levels of measurement and measures of distribution characteristics Level Central tendency Spread Nominal Mode Percent distribution Ordinal Interval/Ratio Median Mode Mean Median Mode Minimum/Maximum Range Percentiles Percent distribution Variance Standard deviation Minimum/Maximum Range Percentiles Percent distribution

15 Measures of central tendency Level Definition Example Mode Median Mean The attribute of a variable that occurs most often in the data set The value of the middle case when the cases have been placed in order or in line from low to high The arithmetic mean or average. Computed as the sum of all the valid cases together and dividing by the number of valid cases. Variable = Nationality Mode = Indonesian Variable = Rank (1 st, 2 nd, 3 rd, 7 th ) Median = 5 th Variable = Age Mean = 24.35

16 Levels of measurement determine the possible statistical analyses Nominal Cross-tabulations Chi-square Frequencies INCOME Total Count Expected Count Count Expected Count Count Expected Count Count Expected Count Count Expected Count Count Expected Count Count Expected Count Count Expected Count Count Expected Count Count Expected Count Count Expected Count MARITAL STATUS SINGLE MARRIED Total

17 Levels of measurement determine the possible statistical analyses Ordinal Spearman correlations Non-parametric analyses Interval and ratio Pearson correlations Parametric analyses

18 Measures of central tendency Mode [=MODE(target range)] Median [=MEDIAN(target range)] Mean [=AVERAGE(target range)] Lets try that using the sample data! TIMSS Country_X Grade 8 data.xlsx

19 Measures of spread Standard deviation [=STDEV(target range)] Min [=MIN(target range)] Max [=MAX(target range)] Percentiles [=PERCENTILE(target range, k th percentile)] where k ranges from 0.00 to 1.00

20 Characteristics of a distribution Skewness: a measure of the asymmetry of a distribution. [=SKEW(target range)] The normal distribution is symmetric and has a skewness value of zero. Positive skewness: a long right tail. Negative skewness: a long left tail.

21 Characteristics of a distribution Kurtosis: A measure of the extent to which observations cluster around a central point. For a normal distribution, the value of the kurtosis statistic is zero. [=KURT(target range)] Leptokurtic data values are more peaked (positive kurtosis) Platykurtic data values are flatter and more dispersed along the X axis (negative kurtosis)

22 Measures of spread Frequencies [=FREQUENCY(target range, groups)] where bin ranges are groups that includes values less than and up to each bin value Bins = 10, 20, 30, will result in 4 groups (bin +1) Group 1= less than or equal to 10 Group 2= 11 to 20 Group 3= 21 to 30 Group 4= more than 30 Enter as array formula Write the formula in the first cell of the output range, select output range equal to number of groups, press F2, then CTRL+SHIFT+ENTER) Percent distribution can be computed by dividing Frequencies with total cases [=COUNTIF(target range, value)] Lets try that using the sample data!

23 Practical exercise! TIMSS Country_X grade 8.xlsx Complete the Descriptive statistics for Boys, Girls, and the whole sample Save your results as we will use them in later sessions.

24 Analysing and Understanding Learning Assessment for Evidence-based Policy Making Inferential statistics and Hypothesis testing Bangkok, 14-18, Sept Australian Council for Educational Research

25 Confidence intervals Standard error of the mean The standard error is an indicator of how precise the statistic is, and how close it is probabilistically to the parameter (e.g., the true mean). Confidence intervals are based on the SE SSSS XX = ss nn CCCC lower = XX ZZ(SSSS XX) CCCC upper = XX + ZZ(SSSS XX) Z=1.96 corresponds to a 95% CI

26 Confidence intervals

27 Confidence intervals

28 Confidence intervals Lets try computing SEs and CIs with data! Means and SDs for boys and girls on Math achievement Standard error of the means Confidence intervals

29 Inferential Statistics Estimating population parameters Inferential statistics can show how closely the sample statistics approximate parameters of the overall population. The sample is randomly chosen and representative of the total population. The means we might obtain from an infinite number of samples form a normal distribution. Source: Johnson, B. & Christensen, L. (2012). Educational Research: Quantitative, Qualitative, and Mixed Approaches. Thousand Oaks, CA: Sage.

30 Inferential Statistics What can we say if we have a sample, and it s confidence interval does not overlap with the confidence interval of another sample?

31 Testing Hypotheses (1) Inferential Statistics Research hypothesis vs. statistical hypothesis Statistical hypothesis testing: comparing the distribution of data collected by a researcher with an ideal, or hypothetical distribution - significance level/alpha (α): e.g.,.05,.01 must be set before testing! - statistically significant means there is sufficient evidence to reject the null hypothesis. - it does NOT mean that the alternative hypothesis is true.

32 Testing Hypotheses (2) Statistical hypothesis testing Making errors in hypothesis testing - Type I error: alpha error, or false positive - Type II error: beta error, or false negative

33 Relationship between α and β

34 Statistical hypothesis testing Our conclusions Not guilty Guilty Reality Guilty Not guilty Correct conclusion Type II error (false negative) Type I error (false positive) Correct conclusion

35 Sometimes, false positive is worse It s better for ten guilty men to go free than for one innocent man to be executed -- anonymous wise lawyer Our conclusions Not guilty Guilty Reality Guilty Not guilty Correct conclusion Freed guilty Condemned innocent Correct conclusion

36 In fire alarms, false positive is better A false negative would REALLY suck. -- anonymous wise homeowner Our conclusions No fire Fire Reality Fire No fire Correct conclusion Burned to death Unnecessary panic Correct conclusion

37 Statistical hypothesis testing Testing for significance Remember: statistical significance is NOT the same as practical significance! In inferential statistics, we are only concerned with statistical significance. Practical significance is a judgment call to be made by the researcher and audience.

38 Statistical hypothesis testing Making errors in hypothesis testing Increase the power of a statistical test Use as large a sample size as is reasonably possible Maximize the validity and reliability of your measures. Use parametric rather than non parametric statistics whenever possible. Whenever we test more than one statistical hypothesis, we increase the probability of making at least one Type 1 error. For multiple hypotheses, a correction (e.g., Bonferroni) needs to be applied to our statistical tests.

39 Statistical hypothesis testing Steps in Hypothesis Testing 1. State the null and alternative hypotheses. 2. Set the significance level before the research study. (Most educational researches use.05 as the significance level. Note that the significance level is also called the alpha level or more simply, alpha.) 3. Obtain the probability value using a computer program such as SPSS. 4. Compare the probability value to the significance level and make the statistical decision. Rule 1:If probability value is less than alpha, reject the null hypothesis. Conclude that finding is significant. Rule 2: If probability value is greater than alpha, fail to reject the null hypothesis. Conclude that the finding is not significant. 5. lnterpret the results. That is, make a substantive real-world decision and determine practical significance. Source: Johnson, B. & Christensen, L. (2012). Educational Research: Quantitative, Qualitative, and Mixed Approaches. Thousand Oaks, CA: Sage.

40 Exercise 1 Statistical hypothesis testing Let s compare two sets of data and test the hypothesis that their means are equal or different Simple T-Test for independent means [T.TEST(data range 1, data range 2)] Data: data check.xlsx Sets to compare: Science scores for sets A vs B, and B vs D

41 Exercise 2 Statistical hypothesis testing Let s compare the science achievement of those parents completed only lower secondary (PHEL=4) vs those whose parents have a university degree (PHEL=1), and test the hypothesis that their means are equal or different Simple T-Test for independent means [T.TEST(data range 1, data range 2)] Data: data check.xlsx Sets to compare: Science scores for sets PHEL=4, and PHEL=1