Statistical Sampling: An Overview for Criminal Justice Researchers April 28, 2016

Transcription

1 Good afternoon everyone. My name is Stan Orchowsky, I'm the research director for the Justice Research and Statistics Association. It's my pleasure to welcome you this afternoon to the next in our Training and Technical Assistance Webinar Series on Statistical Analysis for Criminal Justice Research. We want to thank the Bureau of Justice Statistics, which is helping to support this webinar series. Today's topic will be on statistical sampling, we will be providing an overview for criminal justice research. And I'm going to turn this over now to Dr. Farley, who's going to give you the objectives for the webinar. Greetings everyone. Okay, so the webinar objectives. Sampling is a very important and useful tool in social science criminal justice research, and that is because in many, if not most, situations accessing an entire population to conduct research is not feasible. Oftentimes it would be too time-consuming and expensive. As a result, the use of sampling methodology, if utilized properly, can save time, money, and produce sample estimates that can be utilized to make inferences about the larger population. The objectives of this webinar include: describing the different types of probability and non-probability sampling designs; discussing the strengths and weaknesses of these techniques, as well as discussing the importance of sampling to the external validity of experimental designs and statistical analysis. This slide presents a simple diagram illustrating the process and goal of sampling. Using the appropriate technique, a sample is drawn from the larger and known population, and if this sample is representative of the population we can take what we have learned about the sample and make an inference about the entire population. What is missing from this diagram are the important questions that need to be answered prior to drawing a sample. In fact, the answers to these questions will likely impact the type of sampling method that is selected. For example, what is the nature of the study? Is it exploratory, descriptive, or analytical? What are the variables of interest? What is the target population? What is the data collection mode? Is it survey, or interviews? What is the sampling frame, and what is the target sample size? Besides thinking about these keys questions, it is also useful to be familiar with the terminology associated with sampling. So let's take a moment to review a few key terms. Page 1 of 13

2 Target population is the collection of elements about which we wish to make an inference. While we often speak of populations as very large, hence the need for sampling, target populations can be quite small. So in one case the citizens of California may be the population of interest. In another case you may be interested in a very specific population, say 16 to 18-year-old incarcerated females with a history of heroin addiction. Unit or case refers to the elements you are interested in. These elements could be things like people, organizations, or documents. Sampling frame is the list of all units of the population of interest. For example, if your target population was a local university's students, then your sampling frame would be all students enrolled at the local university. For another example, let's say you want to survey residents of Maine. You could use voter registration records and/or the phone book as your sampling frame, but I'm sure many of you realize that voter registration and phone books are imperfect and incomplete lists. More resources can be added in an attempt to cover more of the population. For example, in addition to voter registration and phone books one could add DMV records. But even then this would still exclude people, and this highlights a common challenge. In almost all cases the sampling frame will not perfectly match with the target population, and this leads to errors of coverage. This error, referred to as sampling error, means that the sampling does not accurately represent the population. Other ways that this may occur is through non-response. This could be volunteers selectively refusing to participate or participants refusing to answer particular questions. Understanding sampling error is extremely important when working with sample estimates and attempting to infer to the larger population, and Stan will discuss this later in the presentation. Okay. So sampling techniques that we are going to discuss today. There are two general categories of sampling techniques: probability and nonprobability sampling. With probability sampling every element has a known chance of being selected into the sample. This process eliminates selection bias, allowing for the sample estimates to be generalized to the larger population. The specific sampling methods that fall under probability sampling that will be discussed today include: simple random sampling, stratified sampling, systematic sampling, and cluster sampling. With non-probability sampling every unit of the population does not have a known probability of being included in the sample. Subjects are Page 2 of 13

3 selected based on the subjective judgment of the researcher. As a result, this type of sampling methodology is vulnerable to selection bias and sample estimates that cannot be generalized or inferred to the larger population. Examples that we will be discussing today include: convenience, judgmental purposive, snowball, and quota sampling. To begin with simple random sampling. In simple random sampling every unit of the population has the same known probability of being included in the sample, and the sample is selected from the population randomly. In other words, each individual in the population has an equal opportunity to be selected for the sample. Say there are 1000 people in your sampling frame. For example, utilizing a book of names and numbers, going back to the college example, and you have 1000 people in your sampling frame. So you've identified your sampling frame, and you would then number each element in the sampling frame from 1 to n, and in this case, this example, it would be 1 to Then you would utilize a chance mechanism to assist in selecting the random sample. Here I have listed a web resource called the random number generator. Advantages of this method is that it is very simple and it relies on probability theory, and it reduces bias. However, this method is also very expensive and time-consuming, and is not feasible in many circumstances. But to provide you an example, say you work for a research agency interested in conducting a study of non-violent felony offenders diverted to drug court for treatment of a heroin or opiate addiction. A very specific population. You want to follow-up with and interview participants who fit this criteria, and who graduated in The total number of graduates is 50, and let's say you're interested in sampling 20. The courts have provided you with a list of offenders, how do you pick your 20? First you must number your graduates from 1 to 50, and then go to the random number generator, for example, this is what... the link that I provided, it will send you to this page, and then you fill this site out. So for example, you answer this list of questions. How many sets of numbers do you want to generate? You just want to generate one. How many numbers per set? 20, since you're interested in selecting 20. What is the number range? 50 graduates equals 1 to 50. Unique value? Yes. And then, generate? This could be whatever is the personal preference, I Page 3 of 13

4 usually choose least to greatest. And then you press on, randomize now, and it will produce a list of numbers. So I received a random list of numbers, and this resulted in 1, 3, 7, 9, so forth, popping up, and this is my random sample of 20 individuals, 20 graduates. Okay. So moving onto systematic sampling, this involves the selection of elements from an ordered sampling frame. So it is a variation of simple random sampling, and it is often considered more practical and easier to use in field settings. Using this procedure every element in the population has a known probability of selection, but not every element has an equal chance of being chosen like simple random sampling. In systematic sampling, it is best used when given a population that is logically homogeneous because systematic sample units are uniformly distributed over the population. You're using a constant interval to select your sample. So a researcher needs to make sure that there is no hidden pattern, as this would threaten randomness. So here is an example, again. Say you wanted to conduct a household victimization survey in a town. Now let's say there are 40 houses in this very small town, and you want to sample 10 of them. First, you would want to establish your selection interview. You would do this by dividing 40, the number of houses, by 10, which is the sample you are interested in, and the result is 4. This means that you are going to sample every fourth house. But the next question is, where would you start? One way to do this is to randomly select a number between 1 and 4. Let's say we randomly selected 3, that means that you would start at the third house and then sample every fourth house after that. So there's the third house we begin with, and then we begin to sample every fourth house until we get this done. Now remember to be aware of possible patterns. Let's say every fourth house in the survey that we've selected happens to be a fraternity or a sorority house, or happens to be a halfway house, this will certainly bias your results. Okay. So continuing on to the next example, stratified random sampling. This is a useful method if a subgroup of interest makes up a small portion of the overall sample. For example, if you are interested in examining the racial and ethnic differences in victimization random sampling may not be the appropriate technique as some racial groups are very small, and the Page 4 of 13

5 use of random sampling may lead to smaller groups either having a very small number of cases in the sample or being left out completely. There are two types of stratified random sampling: proportionate and disproportionate. With proportionate sampling the size of the sample selected from each subgroup is proportional to the size of that subgroup in the population. The drawback for this is that you will still have an over or under-representation of certain subgroups, and this is where disproportionate sampling serves as a useful tool. With disproportionate sampling the size of the sample selected from each subgroup is disproportionate to the size of that subgroup in the population, because you are intentionally oversampling these smaller groups. And this allows for the rare subgroups to be more represented in the sample. Disproportionate samples ensure that you will have enough members of all groups in your sample. So to provide an example, here are four racial and ethnic groups, Caucasian males, African American males, Latino males, and Asian males. And here, overall, that this is our population, we see that Caucasian males represent 50% of the group, African American males represent 25%, Latino males represent 15%, and Asian males represent 10%. So if we were to utilize proportionate sampling we would randomly select a sample from each of these groups, but then they would represent the same percentage. So if we wanted to, for example, interview 20 individuals, that would be 20 individuals... for 50%, if we are focusing on Caucasian males, that 50% of 20 is 10. So we would recruit 10 Caucasian males. 25% of 20 for African American males would be 5, 15% of 20 for Latino males would be 3, and 10% of Asian males is 2. So let's see here. So here is a random selection of each group utilizing proportionate sampling. So they still, again, represent Caucasian males 50%, Latino males 15%, that 3, African American males the 25% sample, and the Asian males the 10% sample. But disproportionate is the alternative. So say we want to oversample some of the smaller subgroups. Here is an example where we would disproportionately sample 25% from each group, and that would represent 4 Asian males, 4 Latino males, 8 Caucasian males, and 4 African American males. So now they all represent 25% of the sample. So previously, for the smallest group that we have, Asian males, prior, utilizing the proportionate sampling methodology, we only recruited two for the sample. And utilizing this methodology we now have recruited 4. Page 5 of 13

6 And this is a way that can be altered, and the decision on how to pull and what percentage of each population, the percentage for the sample that you want to pull, is a decision that is determined by the researcher and the purposes of the research. The next example is cluster random sampling, and this may be useful when attempting to sample a larger region, like a state, especially when one has limited resources. For example, if you are interested in conducting a statewide, in-person household victimization survey, because this would not apply if you were utilizing phone or internet survey avenues, random sampling would not be feasible because the population is so dispersed across a wide geographic region that you would have to cover a lot of ground to get to each of the units in the sample. However, in cluster sampling, it utilizes a different methodology that eases the pressure on time and money constraints. So with cluster sampling you would divide the population into nonoverlapping areas, oftentimes concentrated in natural clusters. This could be city blocks, this could be schools, this could be hospitals, counties, etc. Then you would randomly select a subset of clusters to sample. This would be the first stage of sampling. Cluster random sampling can be a multistaged process where you continue to a second, a third, and a fourth stage depending on, again, your research and your research purposes. So I will provide an example. So say you are interested in conducting an in-person interview with middle school students regarding bullying or school victimization. Utilizing Kansas as an example, because it has 105 counties, say you would like to randomly select 25%, which is approximately 26 counties. Using the random number generator, say you list your counties alphabetically and then number them from 1 to 105, use the random number generator to then produce 26 numbers that help you identify the counties that will be selected. And I did this, and here are the 26 counties that were selected from my random selection process. So that would be phase one. Here, for cluster sampling, you can continue to a second stage. This entails randomly selecting schools out of each of the counties that were selected in stage one. For example, let's take Crawford, and say we randomly selected 16 of the 40 middle schools in the county, and I'm just making up the number of 40 in case there is somebody who actually is familiar with Crawford County. But say you were interested in selecting 16 of the 40 middle schools, the number of Page 6 of 13

7 schools may depend on the class size, because the smaller the school the more you will want to sample. But for this example we are selecting 16. Now you can even go further. So you have selected your 16 schools, you can go further to a third stage of sampling where you could randomly select students out of each of the randomly selected schools from stage two. Or, you could work to sample and recruit all the students from every single school that you have randomly selected. So you could keep doing this type of sampling until you achieve your goal. Alright, moving on to non-probability sampling methods. The first one to discuss is convenience sampling. Convenience sampling is referred to as accidental or haphazard sampling. It is considered quick, convenient, and economical, however, utilizing this type of sampling prevent somebody from inferring to the larger population. And it is one of the most commonly utilized methods of sampling. Examples of this type include: man on the street samples; the use of college students in psychology research, which I'm sure everybody is really familiar with; as well as volunteers. Purposive sampling, also referred to as judgmental sampling, usually entails having one or more specifically predefined groups in mind for recruitment. Purposive sampling relies on the judgment of the researcher when selecting the units to be studied, and usually the sample size is quite small. The goal is to focus on a particular characteristic of the population that are of interest. Now, this type of sampling is useful when you need to reach a targeted sample quickly, and where sampling for proportionality is not the primary concern. With purposive sampling you are likely to get the opinions of your target population, but you are also likely to overweight subgroups in your population that are more readily accessible. Two types of purposive sampling can be typical case sampling, which is pretty easy to interpret, it's just selecting the cases that fit your typical characteristics, and on the other end, you could also sample for extreme or deviant case sampling, which is selecting for unusual or special cases. Other types of purposive sampling include: maximum variation sampling, homogeneous sampling, total population sampling, and expert sampling. Quota sampling is the next method. First the population is segmented into non-overlapping groups, like stratified sampling, and judgment is used to select the subjects or units from each segment. In other words, the researcher deliberately sets the proportions of strata within the Page 7 of 13

8 sample, and this is usually done to ensure the inclusion of a particular segment of the population. The proportions may or may not differ dramatically from the actual proportions in the population. For example, say a researcher is interested in the attitudes of members of different religions towards the death penalty. In Kansas, a random sampling might miss Muslims, since they are a relatively small percentage of the population. To be sure of their inclusion, a researcher could set a quota of 5% for the Muslim sample, however, the sample will no longer be representative of the actual proportions in the population, and this may limit generalizing to the state population. So snowball sampling is another example, and this is utilized when subjects are very hard to find or a very small population. An example would be trying to also access a vulnerable population, for example, research that is attempting to access juvenile prostitutes in large cities. One way to access that vulnerable and small population, oftentimes hidden population, is to utilize this methodology. Here, you would recruit one person and give them, for example, three cards to hand out to three people who fit the study sampling interest. If each one of them returns for an interview they will each then be given three more cards to hand to people that they know, again, who fit the criteria. And it continues so on and so forth. Another type of this sampling, it's often called respondentdriven sampling, and it continues until you have either saturated the sample, recruited as many individuals as you can, let me go back for a second, or you have reached your goal. And so this is the last sampling method. And so I am now going to turn it over to Stan, who is going to start by talking about sampling and standard error. Thanks, Erin. So I wanted to talk about a couple of special topics, if you will, that utilize these concepts that Erin has been talking about. The first one is the notion of standard error. The standard error can be thought about as the standard deviation of the sampling distribution of a statistic, in this case the mean. So, if we were to, for example, draw random samples of a specific size, whether that's 100 or 500 or 1000, and draw an infinite number of those samples, that distribution... each of those samples would have, let's say, a mean, and that mean of that large, infinite number of samples would be equal to the population mean, and the Page 8 of 13

9 standard deviation of that distribution of samples would actually be equal to the standard error. So you can imagine that distribution, that theoretical distribution, would be normally distributed, mean equals population mean, standard deviation equals the standard error. So for a mean, for example, the standard error would be the standard deviation divided by the square root of the sample size. And what that is, that standard error then becomes an estimate for how far the sample statistic, so for the mean and for your particular sample, is likely to be from the population, in this case, mean. And I'm sure most of you recall that one of the ways that those standard errors are used is in terms of developing a confidence interval for estimates that are derived from samples. So for example, for the mean the confidence interval will give us an estimate of where the population parameter will fall. So rather than having a point estimate, which is the sample mean, the confidence interval bounds that so that we have an upper and a lower limit for that sample statistic, which in this case is the standard error of the mean that we just calculated multiplied by a z- score. Added to the mean it gives you the upper limit, and subtracted from the mean gives you the lower limit. And, again, this estimates how far the sample statistic, in this case the mean, is likely to be from the population's mean. The z-scores that are typically used in calculating confidence intervals are shown here for a 98%, 95%, and 99% confidence intervals. Here's a depiction of that theoretical distribution, and in that normal distribution 68% of the cases will fall within 1 standard deviation of the mean, or the parameter estimate, 95% will fall within 2 standard deviations, and 99.7% will fall within plus or minus 3 standard deviations. And those are the corresponding z-scores. So for example, in this season of presidential election coming upon us we have many polls that come out about who folks are gonna vote for, and those are examples of estimates of a population proportion, the proportion voting for one candidate or another. And those are always given with those standard error plus or minus percentages, plus or minus 2%, 3%, and that's how those are derived. The other piece that always comes up in discussion of sampling is sample size. How big a sample size do I need? So this a very basic, elementary view of that question. Sample size, generally speaking, is a function of Page 9 of 13

10 these elements. First, confidence. So the more confidence you want to be in the estimate of the parameter of interest, the larger the sample size is going to need to be. Error rate, obviously, the smaller that the error rate, the greater the sample size. So if your error rate's plus or minus 1% you're gonna need a larger sample than if it's plus or minus 5%. The variability of whatever it is you're measuring in the population. The greater the variability, the greater the sample size. So if you think, well, you don't have to think. If you go back to this, obviously, the more spread out those scores are along that distribution, in order to be able to capture those scores in the tails you're going to need more cases. If you have fewer cases then you're less likely to be able to capture the range when that range is large. When the range is smaller and concentrated, say, within plus or minus 1 standard deviation, you'll be more likely to have sample that's representative of that population with a smaller sample size. And then finally, to some degree, sample size is related to population size. Generally speaking, the larger the population the larger the sample size is going to need to be, although there is a point at which that is no longer true. So, let's take a look at the very basic formula for sample size. This is in a case where we're trying to estimate a proportion. So that sample size will be a function of the confidence level, z, the variance, which is the standard deviation squared, that sigma squared term, and then the error rate, and we'll just use a.05 error rate as an example. So there's the formula. So, as you can see, it's the confidence level times the variance, divided by the square of the error rate. I mentioned that there... Oh, let me just say that there's a separate formula for the sample size for estimating a proportion and the sample size for estimating a mean, and this one's a bit easier because you have these standard terms that you can enter. But in any case you will need some measure of the variance in this case, and in the case of the mean, the same thing. So sometimes you can do that based on previous research, for example, or that sort of thing. I mentioned earlier that the fourth element is the population size to a point, and it turns out to be the case that once you reach a certain point for any given level of confidence, error rate, and so forth, that the sample size will reach an asymptote with regard to the size of the population. So that adding more sample doesn't necessarily get you more Page 10 of 13

11 representativeness in terms of the population size. And that asymptote is roughly around 360, 365, for populations over, say, 10,000. Now this is very, very rudimentary, and obviously doesn't take into account any of the kinds of stratifications that Erin talked about earlier, or the ability to make certain kinds of comparisons. So that, for example, in the election polling, trying to get a representative sample of the country where you can make comparisons between groups, and looking at an error rate of, say, plus or minus 3%, those sample sizes will typically be between 1,500 and 2,000. So it really is... this sample size that I'm talking about is just the basics and just a way to get started. Here's an alternative way to think about sampling and sample size in particular, and that's in regard to research design. So, as we all know the experimental, randomized controlled trials are the gold standard in research design, and they require participants to be randomly... it says designed, it should say assigned, to either a treatment or a control group. And the reason that this is the gold standard is because random assignment controls for a variety of factors, other than the independent variable, that might account for any observed differences between your two groups, in this case your experimental group and your control group. In Campbell & Stanley terms, those of you who are familiar with that, these are known as threats to internal validity. But the other half of what Campbell & Stanley talk about is external validity, which is, basically, the generalizability of a particular study's conclusions to other people, in other places, and at other times. And this is... while a lot of social science research and methods focus on internal validity there's not as much focus on external validity, which really does depend to a large degree on sampling and the method of sampling. And so Campbell & Stanley also talk about threats to external validity, including having a non-random sample, which, as Erin mentioned, is produced by... several of those methods can result in a biased sample. Also, however, non-response, so whatever instrument you're using, or instruments you're using, people's failure to respond. And also dropout, so as your study goes on folks dropping out of your sample, and particularly in a non-random fashion. But there's another way of thinking about sample size, and that is what's known as power analysis. And I'm just going to go over the basic concept here because I think it's interesting, and then if you're interested in this you can delve into it further. So, when we think about internal validity and randomized controlled trials and significance testing, we usually talk Page 11 of 13

12 about, if you remember back to your graduate or undergraduate statistics class, type I error, which is concluding that there is an effect, so in this case a difference between the experimental and control groups, when in fact there's not. And that's the p less than.05 that... and that decimal point is in the wrong place, that should be p less than.05 and not 0.5. The main thing that we worry about when we design research studies is rejecting the null hypothesis when in fact it should not be rejected. So if we have a treatment and we're trying to see if that treatment is effective for, let's say, a new drug or a new counseling method or something, and we want to compare that to a control group, we want to be able to observe a difference between the experimental and control group large enough so that we are pretty sure that a difference of that magnitude could not have happened by chance. And that's what that probability level means. So, by custom, as you know, we've adopted.05 or.01 as being that probability which basically says that five times out of 100 we're going to observe a difference between the experimental and control group of a particular size by chance. So in other words, for an observed difference that big, only five times out of 100 will an observed difference that big be due to chance, and 95 times out of 100 the difference will be due to, in fact, a real difference between the two groups. Or thinking about it another way, the difference between the population that comprises the treatment group and the population that comprises the control group. Type II error, however, is the converse of that. So type II error refers to the idea of concluding that there is no effect when in fact there is. So it's a different sort of a problem, it's sort of like the clouds are a problem, it's finding no significant difference between the groups, but in fact there was a significant difference. Well, how does that happen? One of the ways that it happens... well there's a couple ways that it happens, but how is it minimized? One way is to think about the size of the effect, so how large is the effect that you're looking for? So think again about an experimental group and a control group, how much of a difference... And let's say, let's just make it easy and say we're talking about a drug study rather than using a criminal justice example, so how much of a difference between the groups does that drug make, in terms of the symptomology improvement? How large is that effect size, if you will? So the difference between the two groups. Page 12 of 13

13 And then, the second question then becomes, whatever sample size that I'm using, is it large enough to actually detect an effect of that size? So the smaller the effect size, the larger the sample's going to need to be. The larger the effect size, the smaller the sample is going to need to be in order to detect it. Now in social science research we tend to deal with some pretty small effect sizes, and so this is going to affect how large your sample needs to be, but it's a different kind of perspective on sample size than the one that's provided by the formula that I showed you earlier. So in this question about power, how powerful is my design? Is my design powerful enough? Do I have a large enough sample to detect the effect size that I'm interested in? How do you know how large that effect size is? Well, you have to estimate it, and you have to estimate it from previous studies, for example, that have been done of the same kind of phenomenon that you're studying. So that will give you a sense of how big an effect you're looking for, and there are formulas that you can apply to that effect size that will give you the minimum sample size that you would need to detect an effect of that particular magnitude. So that is our webinar on sampling. We'd like to thank you very much for joining us today. We want to make you aware that this is topic number two in a series of topics that we are planning in the Statistical Analysis for Criminal Justice Research Series. We're going to be perhaps sticking to this order, perhaps not, but we will be covering significance testing, comparing means, comparing proportions, looking at correlation and linear regression, looking at multiple linear regression, looking at logistic regression, and exploratory data analysis techniques like factor analysis, and we'll also be talking at some point about how to effectively display data. So thank you again for joining us, and have a great day. Page 13 of 13