How many speakers? How many tokens?:

Transcription

1 1 NWAV 38- Ottawa, Canada 23/10/09 How many speakers? How many tokens?: A methodological contribution to the study of variation. Jorge Aguilar-Sánchez University of Wisconsin-La Crosse

2 2 Sample size in Sociolinguistics Finding the optimum number of subjects that guarantees validity and representativeness to the sample is still an unresolved problem in sociolinguistics (Silva-Corvalán, 2001)

3 3 What has been assumed Sample size is a matter of numbers e.g., The more, the better Sample size is a matter of incorrect assumptions regarding statistical tests e.g., a fixed number of speakers/tokens per cell

4 4 The origin of these assumptions The belief that to increase the ability to find statistically significant results, we need to increase the sample size regardless of other theoretical and empirical implications. The belief that for a certain statistical test, an expected number per cell is desired. e.g., Chi-square à 5 per cell The belief that samples with an equal number of tokens per variable studied suffice to determine association between linguistic factors and the DV regardless of their presence in the population (Lavandera, 1975).

5 5 What has been done Labov (1966) 25 subjects for every speakers of the population under study Convenience samples As many subjects as we can collect Due to limitations of budget and time

6 6 What should be done Sample size should be determined by theoretical issues such as the nature of the problem and the resources that the sociolinguist has to carry out his/her investigation (Silva-Corvalán, 2001).

7 7 The present study I take Silva-Corvalan s (2001) suggestion and I based my contribution on the assumption that sample size should be based on theoretical and practical issues. I follow what is recommended in the field of statistics for the social sciences for the determination of a proper sample size.

8 8 The goals of the present study To introduce the concern that the practice followed in sociolinguistics may be producing underpowered studies. i.e., they may be yielding generalizations that lead to incorrect conclusions due to weakness in design. To propose a procedure to design more economically effective studies with sound research designs. To propose a modification to the variable rule analysis to address data structure issues that have not been addressed in the design of sociolinguistic studies. This affects the determination of proper sample sizes, and the analyses conducted on the data.

9 9 What is Sample Size? It is one of the most valuable factors to consider when designing a research study (Kelley & Maxwell, 2005). It is related to the power of a study. Because of its relationship to power, it is, in fact, one of the most valuable elements in research design.

10 10 Approaches to sample size planning In the social sciences Through a power analytic framework Power of a study is closely related to its replicability, which results in the building of a body of cumulative knowledge (Cohen, 1988). Power computations are most meaningful when they are done, as part of the study design, before data are collected and examined (Wilkinson & the APA Task Force, 1999).

11 11 Approaches to sample size planning Despite the importance given to the statistical significance of a test, very little attention has been paid to the report of the calculation of sample sizes in the field of linguistics, let alone to the power of each study.

12 12 Power of a study Power is the probability of correctly rejecting the null hypothesis when the null hypothesis is false in the population.

13 13 Importance of the Power of a study A researcher must consider power as a natural aid and an important part of the planning and interpretation stages of research because we aim at discovering important relations between variables (Rossi, 1990). Most empirical research in the social and behavioral sciences is done by formulating and testing null hypotheses which researchers wish to reject as a means of establishing findings about the phenomena studied (Cohen, 1992)

14 14 Importance of the Power of a study Lower power studies can have severe consequences at different levels of generalizability. Low power studies cannot accomplish their central purpose of determining the effects of treatment (i.e., prediction of association) (Murphy & Myors (1998). Lower power studies or studies with small sample sizes are more likely to make Type II errors. Large sample sizes yield almost any result as statistically significant.

15 15 Power Analysis Depends on 3 parameters The significance criterion Alpha The reliability of sample results Sample size The effect size The statistic derived from the statistical test

16 16 The statistical test or Statistical Analysis Determination of statistical significance (i.e., α) and estimation of the probability of error in the statistical conclusion are made within the framework of a particular statistical test. The test itself is one of the factors determining statistical power. Different statistical tests have different statistical power when they are applied to the same data. Power analysis needs to be done during the design of the study.

17 17 The significance criterion (α) The level set for the significance criterion influences the likelihood of statistical significance. A larger alpha makes it easier to reach significance and vice versa (Lipsey, 1990). The probability of mistakenly rejecting the null hypothesis when it is true (α) is a research decision and it is the maximum risk a researcher is prepared to take of making this error (Cohen, 1992). Levels are set at.05 or.10,.010,.0010, etc.)

18 18 The significance criterion (α) Type I and Type II errors

19 19 Effect Size (ES) It is the strength of the relationship between the IVs (X) and the DV (Y) (Vaske, 2002). There are 2 major groups: d- family and r-family d = differences in SD units r = coefficient correlation ES in the social and behavioral sciences tend to be small or moderate One of the reasons why we search for large sample sizes

20 20 Effect Size (ES) It is derived from previous research and/or theory in order to dispel suspicions that they might have been constructed to justify a particular sample size (Wilkinson & the APA Task Force, 1999). It comes from the investigator s knowledge of the field i.e., sample effect sizes found in previous investigations with similar variables The result of pilot studies His/her educated intuition.

21 21 Effect Size (ES) In sum: It is the discrepancy between the null hypothesis and the alternative hypothesis of interest. Every statistical test has its own effect size index. Each standardized ES is a pure and scale-free value that measures the discrepancy between the null hypothesis and the alternative hypothesis or population parameter (Cohen, 1992).

22 22 Structure of the data and statistical tests The recommendation is to use appropriate statistical methods to analyze data. The choice of method is closely related to the research questions and to the structure of the data to be analyzed.

23 23 Problems with the Structure of the data we use The natural structure of the type of data being analyzed does not always meet the assumptions made by the statistical test. Data collected for most sociolinguistic studies are representative of what is called hierarchical or multilevel data.

24 24 Problems with the Structure of the data we use Social Groups 1 Speaker 1 2 Tokens 1 2 1

25 25 Problems with the Structure of the data we use Including two levels of data together bring two problems (Hox, 2002) A statistical one If data are aggregated, the result is that different data values from many sub-units are combined into fewer higher-level values. Much of the information is lost and the statistical analysis loses power. If data is disaggregated, the result is that a few data values from a small number of super-units are exaggerated into many values for a much larger number of sub-units. i.e., we are treating highly correlated observations as independent observations.

26 26 Problems with the Structure of the data we use A conceptual one When interpreting the results, the researcher may commit the fallacy of the wrong level Analyzing the data at one level and formulating conclusions at another level. Ecological fallacy Interpreting aggregated data at the individual level. Atomistic fallacy Formulating inferences at a higher level based on analyses performed at a lower level.

27 27 Problems with the Structure of the data we use Simpson s paradox Group data, drawn from a heterogeneous population, are collapsed and analyzed as if they came from a single homogeneous population (e.g., male and female instances) The question now is, what statistical test to use? Varbrul à GoldVarb (Sankoff, Tagliamonte & Smith, 2005) cannot handle this type of structure. It handles it in separate runs and we can only make comparisons of the results.

28 28 What is available? We need a modification of the Variable Rule analysis through a Multi-level Logistic Regression analysis This type of test allows us to: Learn about treatment effects (i.e., levels of association), Use all the data to perform inferences for groups with small sample size, Predict new cases, Analyze the data that are collected with an inherent multilevel structure, Infer more efficiently for regression parameters, Include predictors at 2 different levels and see their effects on the phenomenon, and Accurately account for uncertainty in prediction and estimation.

29 29 Sample size calculations a = power for overall effect b = power for targeted effect (one specific variable) c = accuracy of parameter estimation for overall effect d = accuracy of parameter estimation for targeted effect (one specific variable)

30 30 Sample size calculations Determining sample size for the study of ser/estar + adjective in the Spanish of Limón, Costa Rica. The parameters set at the onset of the study: Power =.8 Alpha =.05 Statistical test = Hierarchical/Multilevel logistic regression ES = βs Population parameters= 30 tokens average per speaker (Aguilar-Sánchez, 2007)

31 31 Sample size calculations Software used R - an open source software (R Core Development Team, 2009) Package = arm (Gelman et al., 2009) Function = lmer() à linear mixed effects in R Procedure Monte Carlo Simulation It runs the test thousands of times with simulated data. Framework Power Analytical

32 32 Results Model 1 # variables L01= 9 # variables L02=4 Omnibus effect number of observa-ons number of speakers J K

33 33 Results number of observa-ons number of speakers J Model 1 # variables L01= 9 # variables L02=4 Targeted effect = 1 variable K

34 34 Results number of observa-ons number of speakers J Model 3 # variables L01= 6 # variables L02=4 Omnibus effect K

35 35 Conclusions The number of tokens cannot be separated from the number of speakers. Modifications to the statistical model (i.e., variables) after the data is collected affect the power of the study. A hierarchical/multilevel model allows us to appropriately make calculations regarding sample size that can be generalizable to the population under study. Proper sample size calculations allow for the construction of a more cohesive body of knowledge.

36 36 Recommendations Conduct proper power analysis in the design stages of a study to determine the number of speakers and the number of tokens necessary to conduct studies that are generalizable and representative of the population. Report the method used to calculate such sample sizes. Avoid the use of convenience samples or preconceptions about statistical tests with regard to sample size. Modify variable rule analysis to better account for the structure of sociolinguistic data (i.e., multilevel).

37 Thank you very much! 37