PSI R ESEARCH& METRICS T OOLKIT. Writing Scale Items and Response Options B UILDING R ESEARCH C APACITY. Scales Series: Chapter 3

Transcription

1 PSI R ESEARCH& METRICS T OOLKIT B UILDING R ESEARCH C APACITY Scales Series: Chapter 3 Writing Scale Items and Response Options PSI s Core Values Bottom Line Health Impact * Private Sector Speed and Efficiency * Decentralization, Innovation, and Entrepreneurship * Long-term Commitment to the People We Serve

2 Research & Metrics Population Services International 1120 Nineteenth Street, NW, Suite 600 Washington, D.C PSI Research & Metrics 2010 Population Services International, 2010 Contact Information Kate O'Connell 1, Linda Kaljee 2, and Hibist Astatke 3 1. Deputy Director, PSI Research & Metrics 2. Associate Professor, Wayne State University, Pediatric Prevention Research Center 3. Senior Researcher, PSI/Washington For more information, please contact: Hibist Astatke th St. NW. Suite 600 Washington, DC Telephone hastatke@psi.org

3 LEARNING OBJECTIVES By the end of this chapter, the reader will be able to: 1. Write concise and well-crafted scale items 2. Choose appropriate response options for scales 3. Develop instructions for scales BACKGROUND OVERVIEW This chapter outlines the how-to methods for writing scaled items and choosing correct response options. Both of these skills will help you to develop scales with true variation and thereby increase reliability and validity for your multi-item scales. An important component to keep in mind is that writing and developing both scale items and response options will be closely linked to the culture and conceptual meanings of specific determinants of the target research population. These are therefore guidelines to follow, but there is flexibility and all scales and response items must be pilot tested within the target population. In addition, this chapter provides some guidelines for writing instructions to scales. The clearer these instructions are written, the less likelihood of respondents answers not reflecting their actual attitudes and beliefs. Again, good instructions will increase your scale s reliability and validity. 1. GUIDELINES FOR WRITING SCALE ITEMS The following guidelines are designed to assist you when you write your scale items. Some may not apply in a particular situation, and certain issues will differ depending on the language of the scale. The key to writing scale items is that they are short, concise, written in standard language, easy to read and understand, and follow basic grammatical rules of the language in which they are presented. Rules to Write By "What I meant to say was..." Writing survey questions is a very important part of your research design. There are many traps which can be avoided, if you follow some basic guidelines. In this section, we provide some suggestions to help you develop better scaled questions. 1

4 1. Use items with strong opinions It is best if items used in a scale state fairly strong opinions since the response options can then be used to more accurately determine degree of agreement. If an item is too obvious or extreme, most respondents are likely to respond in the same way to the statement so that small differences may not be detected. In the following example, the first item may state the obvious in a community with endemic malaria. The second statement may state an unlikely situation. The third item is stronger and more likely to further differentiate between respondents with perceived moderate versus high severity. Item 1: Malaria is a health problem in this community. Item 2: Malaria is the source of all problems in this community.. Item 3: Malaria is a major health problem in this community. With the first statement, most respondents are likely to either strongly agree or agree. Respondent are likely to strongly disagree with the second statement. Alternatively, those respondents with high perceived severity are likely to strongly agree with the third statement while those with moderate perceived severity are likely to agree or disagree. Thus, the third statement is more likely to better capture true underlying variation or differences among individuals for the determinant severity. 2. Write unambiguous items Items should clearly state one specific idea. Items should be clearly phrased and have a simple meaning as in the example below of items from a scale on Condom Beliefs.. Item 1: Condom use is only necessary with commercial sex partners Item 2: Condoms can be used in loving relationships 3. Write items at a reading level appropriate to the target population Items should be constructed so that they can be easily read and understood by the target population. The items should therefore contain most commonly used words and phrases within the language of the target population. If the items are translated from one language to another (e.g., English/French to the local language), it is very important that the wording in the original language be simple and basic. 4. Write items that avoid jargon or trendy slang While a certain segment of a target population may be familiar with certain jargon or slang, not all members of that population may understand these terms. This does not mean that items are not culturally/contextually specific, and therefore care needs to be taken to ensure that phrases and wording are understood by the general population or at least by the members of the target population. 2

5 For example in Cambodia, clients of commercial sex workers refer to going to sex workers as let s go for a walk. Therefore, this might be an appropriate phrase to include in your scale if your survey is designed for clients of sex workers, but would not be appropriate for a general target population 5. Write in everyday language Items should avoid use of internal jargon, abbreviations or acronyms for products and services. Also, remember that if you are writing about brand attributes of a product, you must make sure that your audience knows what the product is before answering the questionnaire. In some instances, it might be necessary to introduce a product prior to the questionnaire. In the following example, in item 1 the use of ITN may not be understood by the target population. In item 2, this same statement has been written to include the full term insecticide treated nets Item 1. Use of ITNs is a reliable method to prevent malaria. Item 2. Use of insecticide treated nets is a reliable method to prevent malaria. In some instances, it may be necessary to explain a term to make sure that you and the respondents have the same understanding of the term s meaning. For example, when working on HIV and condom use, it may be necessary to explain what you mean by the term condom use. 6. Avoid long complicated sentences Avoid long statements that have information that is not central to the determinant being measured and make it difficult for the interviewer to respond. For example, Condoms are acceptable to use with partners that I know and meet regularly when I go out and socialize. 7. Avoid having more one or more idea in a statement Long statements that have many meanings make it difficult for the respondent to reply. These statements are called double barreled items. A respondent may agree with one idea within the item but disagree with the other. If a statement contains more than one meaning than during analysis it will not be possible to know what aspect of the item a respondent did/did not agree with and therefore increase risk of misinterpreting results. Item 1. Mosquito nets are important to use all of the time and they are reliable. Item 1 contains two ideas: The respondent thinks nets are important to use and he/she thinks they do what they are designed to do (they are reliable). This item should therefore be divided into two separate items for a scale on attitudes toward ITN use. Item 1. Mosquito nets are import to use all of the time. 3

6 Item 2. Mosquito nets are reliable. 8. Avoid double negatives What is a double negative? A double negative is a phrase in which two negative terms or prefixes ( not, no, un- ) are in the same sentence. Double negatives can be confusing for respondents, and can lead to inappropriate answers. An example of a double negative is: Item 1. It is unnecessary to not use condoms with a girlfriend. This statement should be reworded in positive phrasing. Item 2. It is necessary to use condoms with a girlfriend. 9. Avoid using negative terms to reverse the direction of an item Sometimes we want to reverse the wording so that we do not have the respondents repeatedly giving the same answers, e.g., strongly agree. To do this, we often include some items within a scale that we anticipate individuals will agree with and other items that we expect these same individuals will respond to negatively (disagree). For example someone who agrees to the statement that Condoms are safe is likely to also disagree with the statement that Condoms are not safe. Some respondent may however find it difficult to decide whether or not they agree or disagree with this statement. To avoid such difficulties and ensure the reliability of responses given, this item can be rewritten as Condoms are dangerous. 10. Watch for leading questions Avoid biasing the survey with questions that use adjectives and can bias responses. Biased questions can lead to false positive responses. Item 1. "Men who go to sex workers are at serious risk for HIV Item 2. Men who go to sex workers are a risk for HIV 11. Use consistent wording The use of similar phrases for the text of the survey can unify your questionnaire. For example, items can be set up with a single phrase which is used to lead off each item. For example: The follow questions ask you to agree or disagree with the following statements Condoms are inexpensive... Condoms are available at local community health centers... Condoms are easily accessible at pharmacies... 4

7 12. Use time frames that are easy to identify. Also be consistent in wording of time frames. It is important that scaled items are asked within a specific time frame. You may want to know how a respondent has felt during the last month, three months, six months or one year. It can also be helpful to anchor the time period in relation to a local event, holiday, or season. Item 1. During the past 3 months (since the beginning of the dry season). You should also be consistent in wording of time frames. For example, if in one item you ask about one month, do not switch to asking about 30 days in another item. 2. GUIDELINES FOR DECIDING ON RESPONSE OPTIONS AND CORRESPONDING VALUES While many different types of formats exist for scale items and corresponding response options, a Likert scale which is the most common, is recommended for PSIs studies. A Likert scale consists of a statement followed by response options indicating levels of agreement or disagreement with the statement. The following guidelines can be used to help decide the best response options for a scale. 1. Should be easy to understand by the respondents The response options that are provided to the respondents should be clearly stated and should correspond to the determinant in a rational way. 2. Should be consistent with what is being measured. A majority of PSI TraC research study response options will be about agreement. Therefore, options can include: Option 1. Slightly agree, Moderately agree, Incline to agree, Agree very much Option 2. Strongly agree, Agree, Disagree, Strongly disagree In other cases, it is important to determine what you are interested in measuring. For example, are you interested in whether respondents agree with the items, are satisfied with services provided, or expectations have been met? Each of these scenarios will result in different response option labels and need to be considered when developing the scale. 3. Should discriminate between respondents perceptions We want to make sure we can tell the differences between people who have very strong views and those that do not have strong views. If our response options do not adequately discriminate between these respondents, we won t be able to understand the relationship between determinants 5

8 and engagement in behaviors. This increases the risk of collecting data that does not reflect the true relationship between the proposed determinants and behavior. 4. Options should be easy to interpret The wording of options needs to refer to discreet categories, which are clearly understood and recognized by the respondents. If respondents are unable to differentiate between two responses, they may randomly pick one but it may not reflect their true response. For example, is there an understanding in the cultural context between moderately agree and incline to agree. It is also very important that these subtle differences are translated correctly into the local language. 5. Responses should be balanced between positive and negative options Whether an odd or even number of points are included on a scale, it is important that the number of positive and negative points are balanced. For example, if scale points are written to offer more positive options than negative options, the data may be positively skewed data. In addition to the above guidelines, there are other issues in relation to response options that need to be considered during scale development. Number of Response Points to Use There is not one perfect scale for all uses. Instead, you must consider the objectives of your research, and the cultural context where you work. In survey research, scales commonly range from two to ten points. The number of points for your scale should be determined by how you intend to use the data. Although seven to ten point scales may seem to gather more discriminating information, there is a debate whether respondents actually discriminate carefully enough to make these larger-range point scales valuable. In addition, since many times these scales are collapsed into three or five categories for analysis and reporting purposes, the detailed and greater discriminatory characteristics are often lost.. Alternatively, two and three point scale offer little discriminative value and are rarely recommended for multi-item scale research. Generally, four and five point scales are recommended and used in a majority of research. The overarching concerns when choosing the number of response options is the effect on the scale's reliability and ability to discriminate between degrees of the respondents' perceptions of an item (discriminability). There has been a great deal of research devoted to this topic, but disagreements still abound. In general, researchers agree that at least three points should be used (Cohen, 1983) and up to 9 points can be used effectively by respondents (Bass, Cascio, & O'Connor, 1974). However, beyond that, conflicts arise. For example, research summarized by Chang (1994) indicates that over the years researchers have suggested a range of number of points from 3 to 7 to maximize reliability, while other research has indicated that reliability is independent of the number of scale points. 6

9 While this research suggests inconsistent findings, there are issues to consider when determining the optimal number of responses to use. Respondents knowledge of the subject matter. If you suspect a lack of respondent knowledge, this may lead to an abuse of the endpoints of larger point scales resulting in lower reliability than with those with few response points. Consequently, it is worth considering how familiar your potential respondents are with the issues you are addressing in your scales. Similarity or lack of similarity of respondents frames of reference for the issues addressed in the scales.. Specifically, using more response option points may introduce error when a respondent group has very different frames of reference. The larger ranges of points provide respondents with more opportunity to introduce their own unique frames of reference. One final issue regarding number of response option points is that it is extremely important that the same number of response options are used for baseline and subsequent follow- up studies. This consistency across data collection points is essential in order to ensure the comparability of data. Including a Neutral Point When addressing this issue, it is worthwhile to wrestle with a philosophical issue and first determine whether you would like to allow respondents the opportunity to be neutral on the topic. Arguments exist for both including and not including a neutral point, and it is reasonable to ask what effect adding a neutral point has on the responses you receive. Response options without a neutral response, or even numbered scales (4, 6, 8 point scales), can more effectively discriminate between agree or disagree statements because there is not a neutral option. However, this clear division may cause hesitation for respondents who are neutral in regard to a survey item. Without a midpoint option, respondents often choose a positive response, creating positively skewed data. Carefully consider whether a clear division between positive and negative responses is necessary, or whether a midpoint will be more appropriate for your information needs. According to some research, response options with a neutral point have the effect of reducing the percentage of positive responses. As a result, this research suggests that using a 5-point scale with a neutral point would result in lower scores than a similar 4-point scale without the neutral. Again, this is because respondents may be more inclined to choose a slightly positive response over a slightly negative response. However, it is important to consider your respondents when making this decision. Is it likely some will be neutral on the topic? If so, then forcing them to one side may alienate them and result in fewer completed surveys. PSI TraC scale studies usually use 4-point response options (strongly agree, agree, disagree, strongly disagree). In PSI, it is only suggested that a neutral point be added in situations where researchers believe that a large percentage of the population is not likely to give a neutral response in order to avoid making a choice, because of discomfort answering items, or because they don t understand the statement. If a large number of respondents provide a neutral response, true variation will not be captured. 7

10 Selecting Scale Anchors Another issue to consider are what anchor labels should used for a particular scale. Like response option points, choice of these labels can potentially affect the level of discrimination and reliability. As previously discussed, the terms associated with points need to be consistent with what is being measured and therefore not all labels will fit with all types of scales. Three common types of response options are agreement, frequency and evaluation (Spector 1992). Agreement is the most flexible and can used to ask about attitudes, beliefs, perceptions, personality and the environment. Evaluation response option labels can be used for attitudes or evaluating performance. Frequency are best for describing characteristics of the environment related to how often an event occurs, or personality scales that gather information about how often respondents engage in particular behaviors. Bass and his colleagues conducted some early research in this area and provide a table summarizing a variety of response anchors to use for a number of different scenarios (1974). Two examples of labels for 5-point scales which their research has shown to discriminate well are shown below: Scales of Frequency: Never, Occasionally, Fairly/Many Times, Very Often, Always Scales of Amount: None, Some, Quite a Bit, An Extreme Amount, All Additional response anchors are available and can be used to fit a variety of situations. There is no one right scale to use for all occasions, and as usual determining your information requirements ahead of time will help you to make a sound decision. As noted PSI TraC studies will in most situations use agreement labels. These four-point scales along with other examples of labels for specific situations are listed below. Amount Amount Evaluation Evaluation Agreement Agreement Far too much Much higher One of the best Very good Slightly Strongly agree Too much Higher Above average Good Moderately Agree About right About the same Average Fair Inclines to Disagree Too little Slightly lower Below average Poor Very Much Strongly disagree Far too little Much lower One of the worst Very poor 8

11 Ensuring Consistency Across Scales Both the number of response option points and anchor labels should be the same across scales to the greatest degree possible. Obviously, some scales may require different anchor labels because what they measure differs, e.g., attitudes versus evaluation. Inconsistent response options can confuse respondents. The following guidelines should be used for response options across scales on a questionnaire. Keep the same number of points across scales. For example, do not change from a fivepoint to a seven-point scale of agreement. Maintain the same order relationship between points and anchor labels. For example, if '5' is high on the first scale, don't make '1' high on the next scale. Use similar wording for the anchors. Group like items together within the questionnaire. If you do need to change response option labels, separate the scales in different sections of the questionnaire. Reverse coding When using a Likert scale in PSI questionnaires each response option is given a value from one to four. In addition to the above guidelines, PSI recommends having higher scores represent more favorable or positive beliefs, attitudes, social norms etc.. For example, if a respondent s agreement with a majority of the items for the determinants you are measuring is the desired (positive) response, e.g, high self efficacy, we would assign the following values (strongly disagree=1, disagree=2, agree=3, strongly agree=4). In this example, for those items that are negatively phrased, we would reverse scores during the analysis stage before creating a composite score. In this way, respondents with stronger positive beliefs, attitudes or social norms will automatically have higher scores. The following example shows two scales. In the first scale on condom accessibility, all items are worded in a positive direction. In the second scale, highlighted items are worded in a negative direction, and it will be necessary to reverse score these items prior to calculating composite scores. However, the same response option points and anchor labels are used for both scales, and respondents with desired positive selections will have higher scores. 9

12 Strongly Agree Agee Strongly Disagree Disagree Condoms are easily available at all times of the day I can always find a condom when I need one Shops nearby here always have condoms for sale I have convenient condom sources within ten minutes from my place of work Condoms are easily available at all times at night Source: Population Services International. Uganda (2005): HIV/AIDS TRaC study Evaluating Condom Use among Female Sex Workers in Kampala and Kasese. Washington, DC. Condom use is normal in Sweet Heart relationships these days Condom use is normal in spousal relationships these days It s acceptable for a woman to use a condom with her spouse It s acceptable for a woman to use a condom with her Sweet Heart It s acceptable for a man to propose condom use to his spouse It s acceptable for a man to propose condom use to his Sweet Heart It would be strange to use condoms with your Sweet Heart nowadays It would be strange to use condoms with your spouse Strongly Agree Agee Strongly Disagree Disagree nowadays Source: Population Services International. Cambodia (2005): HIV/AIDS TRaC Study Examining Condom Use among Sexually Active Men with Sweethearts in Phnom Penh. First Round. Washington, DC. (modified) - these items need to be reverse scored prior to calculating the scale s composite score 10

13 3. GUIDELINES FOR DEVELOPING INSTRUCTIONS For each scale, instructions need to be provided. These instructions should include the following information: A short (one/two sentences) of the purpose of the scale. This should be in simple concise language, but designed to help direct the respondent to what he/she should be thinking about while reading the items. This may include specific directions regarding timeframes (in the past 3 months) or who (children under age of five years) a respondent should be considering in answering the items. Directions about how the information will be delivered. This will include information about who will read the items and who will write the reponses. I will read the following items to you ; or, Please read each item carefully.. Information about the response options. This will be an overview of the option anchor labels and if necessary a definition of those option anchors.. Information regarding what the respondent needs to do. This will be one or two sentences which will tell the respondent what he/she needs to do in order to answer each item. The following is an example of instructions for a scale regarding attitudes toward ITN. We would like to learn more about your opinion regarding the use of mosquito nets in your household. I am going to read some statements to you about mosquito nets. I would like you to first tell me if you agree or disagree with the statement. I will then ask you if you strongly agree (disagree) or just agree (disagree). Please provide me with the response which is closest to how you feel about the statement. Some scales may need additional information about specific judgment tasks. For example, if a mother is being asked about her perceptions of susceptibility to malaria for children under five years. We would like to learn more about your opinions in relation to malaria. I am going to read some statements to you about malaria. This set of statements is about children under the age of five years. Please think about children under five years of age as you respond to each statement I read. I will then ask you if you strongly agree (disagree) or just agree (disagree). Please provide me with the response which is closest to how you feel about the statement. 11