SURVEY RESEARCH MSc Economic Policy (March 2015) Dr Mark Ward, Department of Sociology
Overview Who to ask Sampling Survey error How to ask Part I Survey modes Part II Question and questionnaire design Questionnaire design Formats for questionnaire items
The process Measurement Concepts Indicators Indicators in survey research Multiple indicators Likert scales Developing indicators
Concepts Bulmer, 1984 Concepts are categories for the organisation of ideas and observations de Vaus, 2002 Abstract summaries of a whole set of behaviours, attitudes, and characteristics which we see as having something in common Bryman, 2012 a label that we give to elements of the social world that have common features and strike us as significant Often, concepts have no set meanings, even within the same academic discipline Necessary to define them and clarify how you define them How have others defined a particular concept? Review literature
Definition of social class A large category of people within a system of social stratification who have a similar socioeconomic status in relation to other segments of their community or society. A social class is not organized, but the individuals and families who compose it are relatively similar in educational, economic and prestige status. Those who are classified as part of the same social class have similar life chances. Some sociologists regards social classes as being primarily economic in nature, whereas others tend to stress factors such as prestige, style of life, attitudes, identification, etc. (A Modern Dictionary of Sociology, pp. 384/5)
Deciding on a definition Nominal definition Working definition in your research Operational definition How it is translated into measurable indicators Operationalisation
Descending the ladder of abstraction Moving from the broad to the specific and from the abstract to the concrete (i.e. questionnaire items.) Operationalisation : Clarifying/defining concepts Specify dimensions and subdimensions of a concept Develop measurements
Latent constructs theoretical in nature; they cannot be observed directly and, therefore, cannot be measured directly either. To measure a latent construct, researchers capture indicators that represent the underlying construct. Latent construct Observation Observation
A measure Vs. an indicator Measures things that can be counted / quantities e.g. age, income, number of siblings, how you voted Indicators a means to measure more abstract things e.g. life satisfaction, happiness
Indicators in survey research In survey research, indicators are generally questions: Simple question (with open or closed answers) Multiple questions (battery of item) Rating (e.g. Likert scale) Sometimes, tests are used (e.g. political knowledge; language ability etc.)
Example of a test
Asking questions
Two conflicting demands 1) The longer the questionnaire the lower the response rate and data quality. These are generally essential for a successful survey. 2) You only have one go at it you cannot ask for further information once the questionnaires are returned.
Types of questions Factual questions Attitudinal questions/beliefs Knowledge questions
Have reasonable expectations participants It is generally not useful to ask questions that respondents cannot be expected to know e.g. Do you think prices will go up next year? This also applies to question about other people: A person cannot generally know another person s attitude for example They may however know some factual information When might it be necessary to ask proxies?
Factual questions About the respondent or about others How old are you? What is the highest level of education your partner has? May include behaviours How often did you go to church in the last month? May rely on respondents memory What can you realistically expect a respondent to remember? At which level of detail?
Attitudes and beliefs Often statements rather than questions Multi-item scales Attitudes E.g. Most people are helpful and can be trusted. Beliefs What percentage of this country s population is foreign born? Norms and values Family is important to me.
Example: Schwartz Human value scale
Schwartz human values scale Nominal definition: Human values are concepts or beliefs about desirable end states or behaviours that transcend specific situations, guide selection or evaluation of behaviour and events, and are ordered by relative importance. Operationalisation: Dimensions: Conformity Tradition Benevolence Universalism Self-direction Stimulation Hedonism Achievement Power Security
Indicators Schwartz human value scale Now I will briefly describe some people. Using this card, would you please indicate for each description whether that person is very much like you, like you, somewhat like you, not like you, or not at all like you? Followed by different items
Example of items used Conformity He believes that people should do what they're told. He thinks people should follow rules at all times, even when noone is watching. It is important to him always to behave properly. He wants to avoid doing anything people would say is wrong. Power It is important to him to be rich. He wants to have a lot of money and expensive things. It is important to him to be in charge and tell others what to do. He wants people to do what he says.
Questionnaire design
General rules for question design Research question rules!!!! What do you need to ask to answer your question? Link concepts of your research question to indicators you are using What can you safely exclude to shorten the questionnaire? Be clear on what you mean Ambiguity! E.g. How often do you visit the doctor?
A better way How many times have you visited a GP for medical treatment for yourself in the last 6 months? [ ] NONE [ ] ONCE [ ] TWICE [ ] THREE TIMES [ ] FOUR TIMES [ ] FIVE OR MORE TIMES
Acquiescence The tendency to chose same answer category repeatedly i.e. always agree / disagree Particularly dangerous as it tends to be stronger with less educated respondents education in turn is often a key predictor of outcomes. To avoid randomly order positive and negative statements Randomly order response categories
Social desirability When the respondent tries to impress the researcher is some way, either consciously or unconsciously. Particularly an issue when dealing with issues of people s self esteem (e.g. income). Also very relevant for sensitive or highly controversial questions. More likely in face-to-face than anonymous surveys
1. Refugees should be welcomed to this country. 2. Since Irish people have immigrated for so long Ireland should be open to people wanting to immigrate here. 3. Asylum seekers should be free to work while their application for asylum is processed. 4. The current methods of processing asylum seeker applications are unfair.
Avoid double-barrelled questions Questions that ask about two things at once should be avoided Respondents may agree with one or the other, but not both Here are some examples that fail to do this: I have my own culture and tradition and see the need to learn about the culture and traditions of other people Do you agree with the Pope s view on contraception? Police are biased towards travellers Which party did you vote for in the last general election?
Over- and under-reporting in surveys Over reported Voting Energy conservation Seat belt use Reading (white-collar workers) Church attendance Exercise Sex partners (males) Under reported Racist attitudes Use of illicit drugs consumption of alcohol Smoking Abortion Energy consumption Criminal behaviour Black market income Sex partners (female)
Strategies for sensitive questions Confidentiality assurance increases response rates for sensitive topics (Singer et al., 1995). Self-administered questionnaires usually work better for sensitive questions. Question wording and question order can help E.g. Some people think that immigrants increase crime rates. Do you agree with this view? Ask sensitive questions towards end of questionnaire
More things to avoid Double negatives Should the instructor not schedule an exam the same week a paper is due? (responding no make a double negative) Complex grammatical structure Questions should be easily understood Technical or formal language Do not ask - Do you feel alienated? Instead something like - Sometimes I feel all alone in the world.
Also avoid leading questions Leading questions point respondent in a direction This can manipulate results quite strongly E.g. abortion vs. killing unborn babies vs. ending a pregnancy. How would you rate the performance of legendary Taoiseach Enda Kenny?
Constructing response categories For closed questions: Categories should cover all possible responses (Exhaustiveness/Inclusiveness) Possibility to include other, please specify Categories should be mutually exclusive (Exclusiveness) In some cases, more than one answer may be given Categories should be balanced i.e. as many positive as negative options
No opinion / Don t know categories (1) From the point of view of the researcher having Don t know categories is undesirable as they are often difficult to interpret and analyse. However we obviously want to avoid forcing people to express an opinion which they do not believe in.
No opinion / Don t know categories (2) Researchers differ greatly in their views on this subject but as a general rule of thumb: In face-to-face administration do not offer a Don t know alternative but record one if it is volunteered. In self-completion formats provide a Don t know category (maybe try to make it less conspicuous?).
Formats for questionnaire items
Closed question formats Checklists two or more answer categories Rating scales Likert scale Horizontal rating scale Semantic differentials Scores E.g. out of 10, Feeling Thermometer Ranking E.g. most important, second most important etc.
Answer categories Checklists (Census 2011)
Rating items (1) 1. Likert items It is important that households pay for water usage 1 2 3 4 5 Agree Agree Neither Disagree Disagree Strongly Strongly When I feel uncomfortable at work, I know how to handle it. Agree 1 2 3 4 5 6 7 8 9 10 Disagree Strongly Strongly
Rating items (2) 2. Horizontal rating scales Very similar to Likert items except that the end points of the scale represent attitude positions rather than strength of opinion. The smoking ban should be extended 1 2 3 4 5 6 7 The smoking ban should be abolished
Wong-Baker FACES Pain Rating Scale Point to each face using the words to describe the pain intensity. Ask the child to choose face that best describes their own pain and record the appropriate number.
Rating items (3) Semantic Differentials
Vignettes These are a more specialist format than the others. A vignette is usually a short story describing a hypothetical social situation which the respondent is then asked to judge in one or more ways. Require more preparation than other formats and need large numbers of respondents. Possible to vary details of the story (as each of the underlined above) and observe differences in response.
Example of a vignette Victor Long, a white, employed sewing-machine operator, was convicted of intentionally shooting his friend, Laura, a housewife. The victim required 2 weeks hospitalisation. The offender claims to have been taking drugs at the time. Victor Long was sentenced to 10 years in prison. The sentence given was [ ] MUCH TOO LOW [ ] LOW [ ] ABOUT RIGHT [ ] HIGH [ ] MUCH TOO HIGH (Miller et al. 1991)