Validity and reliability testing of a short questionnaire developed to assess consumers use, understanding and perception of food labels

Transcription

1 (2010) 64, & 2010 Macmillan Publishers Limited All rights reserved /10 $ ORIGINAL ARTICLE Validity and reliability testing of a short questionnaire developed to assess consumers use, understanding and perception of food labels D Mackison 1, WL Wrieden 2 and AS Anderson 1 1 Centre for Public Health Nutrition Research, Division of Clinical and Population Sciences and Education (CPSE), University of Dundee, Ninewells Hospital and Medical School, Dundee, UK and 2 School of Pharmacy and Life Sciences, The Robert Gordon University, Aberdeen, UK Background: Food labels may have an important function in communicating nutrition information and have considerable potential to influence food choice and dietary behaviour. Objectives: To assess the validity and reliability of a short (self-complete) questionnaire designed to measure consumers use and understanding of food labels. Methods: Questionnaire content was determined by gaps highlighted in a literature review of food labelling. Nineteen questions (49 items) assessing frequency of label reading perceived importance of food labels, regularity of dining out, desire to have nutrition information at specific catering outlets and ability to perform nutrition information tasks were formulated and presented on four pages of A4. With the exception of two open-ended questions, all items were presented as closed (field box) structures. Content validity, face validity, item analysis, repeat and internal reliability were assessed. Results: Nutrition experts (26) completed detailed content validity assessment, resulting in high scores for appropriateness, importance and phrasing of questions, although grammar and terminology changes were required. Face validity indicated that the questionnaire was quick to complete (o15 min), easy to follow and comprehensible. Cronbach s alpha scores (internal reliability) for questions with multiple sections ranged from 0.72 to 0.91, indicating good internal consistency. Repeat reliability testing showed Spearman s correlation coefficients ranging from 0.51 to 0.97 (all Po0.001) showing high temporal stability. Item Difficulty analysis indicated that questions (Section C only) were at an appropriate level (with P between 20 and 80% for all items). Item discrimination analysis ranged from r ¼ 0.43 to 0.70, highlighting that items were suitable for inclusion. Conclusions: This questionnaire is a suitable tool for assessing consumers use, understanding and perception of food labels. (2010) 64, ; doi: /ejcn ; published online 11 November 2009 Keywords: food labelling; labels; validity; reliability; questionnaire development Introduction Food labels are recognized as having a pivotal function in communicating nutrition information to consumers and are considered to have the potential to influence food choice and dietary behaviour. For the consumer to make healthy food choices, they must be able to locate, read, interpret and understand the information presented on food labels. Correspondence: D Mackison, Division of Clinical and Population Sciences and Education (CPSE), Centre for Public Health Nutrition Research, University of Dundee, Ninewells Hospital and Medical School, Dundee, DD1 9SY, UK. d.mackison@dundee.ac.uk Received 21 January 2009; revised 16 July 2009; accepted 20 July 2009; published online 11 November 2009 Encouragingly, most consumers claim to read food labels often or sometimes (Cowburn and Stockley, 2005; European Heart Network, 2003). Available literature, however, suggests that while consumers claim to understand the information they are reading (National Institute of Nutrition, 1999; Stuart et al., 2004; Food Standards Agency, 2007), their actual understanding is selective (European Heart Network, 2003; Cowburn and Stockley, 2005). Consumers display little difficulty judging the nutrient levels of products (Food Standards Agency, 2006), yet struggle when asked to perform arithmetical calculations and comparisons of different sizes and types of products (Levy and Fein, 1998; Rothman et al., 2006). Creating food label information that is easily interpreted by consumers requires knowledge of the specific label

2 attributes that consumers read and use. Energy and fat information are the most commonly read nutrient by the consumer (Black and Rayner, 1992; Institute of Grocery Distribution, 1998; Kessler and Wunderlich, 1999; Marietta et al., 1999; Neuhouser et al., 1999; Higginson et al., 2002; Krukowski et al., 2006; Mannell et al., 2006); however, we know little about the consumer s interpretation of this information and how this influences their food purchase decisions (Grunert and Wills, 2007). In a bid to standardize and ease the delivery of nutrition information to the consumer, many countries have adopted mandatory nutrition information on food labels (for example United States, Australia and Canada) (Hawkes, 2004). If successful, the European Union s recent proposal for a Regulation on the provision of food information to consumers (the consolidation and update of the general food-labelling legislation (Council Directive 2000/13/EC of the European Parliament and of the Council of 20 March 2000, 2000) and the nutrition-labelling legislation (Council Directive 90/496/EEC of 24 September on nutrition labelling for foodstuffs, 1990)) would introduce mandatory nutrition labelling on the principal field of vision (front of pack) on all products throughout Member States (EU, 2008). During the consultation and implementation phases of this Regulation, increased consumer awareness of nutrition information on food labels is anticipated as a direct result of media coverage and as labelling changes become prevalent throughout stores. The introduction of a new standardized nutrition-labelling format provides a valuable opportunity for education interventions focused on food labelling to take centre stage. However, before public health practitioners can tackle educating consumers on the new food-labelling formats, it is imperative that we gain a better understanding of the consumer s views on food labels. More information is needed on what information consumers actually use, and how consumers use specific information and form food purchase decisions based on food-labelling information. Answering these questions will provide useful guidance for a tailored food-labelling education campaign that will focus on the key elements pertinent to improving consumers label reading comprehension. Estimated to be worth d36.6 billion by 2012 (Mintel, 2007), the catering sector has a significant function in everyday life in Britain. Twenty-seven per cent of British household s food and drink expenditure is spent on food and drink (excluding alcohol) outside the home (Defra, 2006). Traditionally, eating out was initiated by a social event or celebratory occasion; however, in today s fast paced society, eating outside the home is often a necessity. As dining out increases, it raises the question, should we be seeing the same nutrition and ingredient information when eating out as we do when purchasing products in a retail environment? As calorie information becomes compulsory in New York chain restaurants with 15 or more outlets nationally (New York City Department of Health and Mental Hygiene, 2007), it becomes evident that we know very little about the British consumers desire to have nutrition information available in catering establishments. To date, there has been limited research conducted on the British consumers views on the application of nutrition style information in catering outlets. The aim of this study was to develop a valid and reliable short questionnaire to assess consumers use, understanding and perception of food labels in both the retail and catering environment. Methods Questionnaire development Key areas were identified after a literature review on food labelling and a draft questionnaire was devised. Five sections (A E) were identified: a. Frequency of reading specific attributes on food labels, in which food labels are read, and importance of food labels b. Frequency of using, views on quantity and perceived importance of nutrition information on food labels c. Objective assessment of nutrition label tasks d. Frequency of using specific catering establishments and desire to see nutrition and additional information available in specific catering establishments e. Demographic information The instrument was intended to be completed independently by adults aged 18 years of age and older, and it was essential that the questionnaire was short in length (four sides of A4 paper), quick to complete (o20 min), easy to follow, comprehensible and contained short questions (with language suitable for the target group). Questionnaire readability was assessed using the Flesch Reading Ease Score available on computer software (Microsoft Word 2003) (Microsoft, 2007). Before validity and reliability testing commenced, nutritionists working in the department checked the questionnaire for both content and clarity. Content validity was assessed by experts with a nutrition and dietetic background. Both nutrition and dietetic professionals were contacted through an internet listserv group, the Community Nutrition Group. Using , a PDF version of the draft questionnaire, brief outline of the project and score sheet were distributed to the group. The score sheet listed each question and asked the respondent to score the item out of 10 in relation to appropriateness, importance and phrasing. A column was also provided on the score sheet for other/additional comments to be completed as necessary by the respondent. Responses were collated and the questionnaire was amended as appropriate. Face validity of the amended questionnaire was assessed by short post-questionnaire interviews with 20 adults representative of the population being sampled. Care was taken to ensure that the sample interviewed were of varying education level and socio-economic position. Individuals were asked about the questionnaire instructions, layout and length. Ease of completion, time to complete and question 211

3 212 difficulty were also assessed. Respondents were given the opportunity to ask questions and raise queries with questionnaire items. Reliability testing was assessed using a convenience sample recruited from local business, with the mean time between completing the questionnaire at time 1 and at time 2, 7 days later. Respondents completing the reliability testing had not been earlier involved with either the content or face validity testing. Ninety-seven participants were asked to complete the questionnaire at time 1 and were informed that they would be required to complete a similar questionnaire 1 week later. Correlation analysis was performed on the reliability testing. Each question correlation was required to be 40.5 and statistically significant for inclusion in the final questionnaire. Cronbach s alpha coefficient was calculated to assess the internal consistency of similar questions. Cronbach s alpha values beyond 0.70 were considered satisfactory for inclusion (Bland and Altman, 1997). As a further measure of reliability Item Difficulty Index and Item Discrimination Index were calculated for Section C (nutrition label task questions). The Item Difficulty Index is determined by calculating the percentage of the population answering the question correctly (P-value). For inclusion, Kline (1993) suggests that the P-value should be between 20 and 80%. The Item Discrimination Index was assessed quantitatively by correlating the score for each item with the total score. Item Discrimination Index correlations are considered suitable for inclusion (Kline, 1986). Results Content validity Twenty-six nutritionists/dietitians responded to the request for comments. Of the 26 nutrition professionals, 84.6% had experience of teaching or advising on food labels. With the exception of question 2 ( Where do you read food labels? ), all questions scored highly for importance, appropriateness and phrasing with mean scores above 8.0 for each question. Overall means for importance, appropriateness and phrasing were 8.77 s.d.±1.41, 8.79 s.d.±1.34 and 8.84 s.d.±1.20, respectively. Amendments to the questionnaire included grammatical changes and the addition of explanatory notes in question 1 (with regards to allergen, nutrition and producer information), emphasis on the terms read and use in questions 1, 2 and 4, and the addition of workplace canteen to questions Despite scoring poorly, question 2 ( Where do you read food labels? ) was not removed as it was felt that this question provided a valuable insight into label reading behaviour (pre- and post-purchase). Face validity Twenty adults from various socio-demographic and educational backgrounds completed a short interview after completion of the questionnaire. Questionnaire completion time was between 10 and 15 min with the majority of respondents (80%) completing the questionnaire in o10 min. All participants found the questionnaire layout to be clear and the majority found both the font size and questionnaire length just right. The questionnaire instructions were considered easy to understand by 80% of respondents (with the remaining 20% stating the instructions were neither easy nor difficult to understand ). Despite respondents claiming to find elements of the questionnaire difficult (in particular Section C of the questionnaire), it should be noted that no sections were omitted. Sixty per cent of respondents skipped the frequency question for Other, please specify in question 1, and as a result, this question was formatted (shading and font alterations) to maximize future responses. Four respondents enquired about the meaning of question 4 ( How frequently do you use nutrition information on food labels? ) and as a result the question was amended to include on food labels when buying food. Readability The higher the Flesch Reading Ease Score, the easier the document is to understand. The questionnaire scored a Flesch Reading Ease Score of 64.7, indicating that the questionnaire was of a standard readability level (Flesch, 1948). Reliability Ninety-seven participants, 61 females and 36 males, completed the questionnaire at time 1. Eighty-one participants, 50 females and 31 males, completed the questionnaire at time 2. The mean time in days between questionnaire completions was 7 days, with a minimum interval period of 6 days and a maximum interval period of 9 days. Internal consistency of frequency of using specific label attributes (question 1), frequency of dining at specific catering outlets (question 17) and desire to obtain nutritional and additional information at catering outlets (questions 18 and 19) were assessed using Cronbach s alpha coefficient. Given that the largest response was obtained at time 1 (n ¼ 97) and participants had no earlier exposure to the questionnaire at this point, time 1 data was used to assess Cronbach s alpha values. Cronbach s alpha coefficients were 0.85, 0.75 and 0.91 for frequency of using specific label attributes, frequency of dining at specific catering outlets and desire to obtain nutritional and additional information at catering outlets, respectively, showing good internal consistency. Repeat reliability testing was carried out on the data obtained from the time 2 sample (n ¼ 81). Good temporal stability was shown with Spearman s correlation coefficients ranging from 0.51 to 0.97, all significant at Po Item Difficulty Index analysis was carried out using questionnaires completed at time 1 (n ¼ 97). Five out of the

4 Table 1 Question Item difficulty analysis for (the first) Section C Difficulty (% answering correctly) 213 Locating the calories per 100 g (Question 7) 91.8 Locating the quantity of fat in the package (Question 8) 76.3 Calculating the amount of fibre in half the pack (Question 9) 84.5 Calculating the amount of saturated fat in 200 g of the product (Question10) 79.4 Looking at the sugar content of this product (Question 11) 22.7 Comparing the fat content of two products (Question 12) 82.5 Comparing the saturated fat content of two products (Question 13) 96.9 Comparing the energy content of two products (Question 14) 96.9 eight questions in Section C were answered correctly by 480% of the respondents, indicating that they were too easy for inclusion in the final questionnaire (Table 1). On the basis of this analysis, this section was revised and independently underwent reliability testing. Nutrition colleagues working within the research centre were consulted during the revision of Section C. Reliability testing of the revised Section C was conducted. Participants received only Section C of the questionnaire and submitted their post-code for matching purposes; no other demographic data was collected. Seventy participants completed Section C at time 1 and 51 participants completed the questionnaire at time 2 (with the average completion time between questionnaires, 7 days). Internal consistency was assessed using Cronbach s alpha on data obtained during time 1 (n ¼ 70). For ability to make judgements on nutrition content from available nutrition information (questions 10 16), the Cronbach s alpha was 0.72, indicating good internal consistency. Repeat reliability of Section C was carried out using data obtained for the 51 participants who completed Section C at both time 1 and 2. Spearman s correlation coefficients ranged from 0.57 to 0.94 and were statistically significant at Po0.001, indicating good temporal stability (Table 2). The Item Difficulty Index was re-assessed for the revised Section C. The P-value s highlighted moderate scores for the new questions indicating that they were not as easy as the initial draft and appropriate for inclusion in the final questionnaire (Table 3). With regards to Item Discrimination Index analysis, all items in Section C achieved correlations above 0.20 (Kline, 1986), with r ranging from 0.43 to 0.70 (Table 3), indicating that these items were suitable for inclusion in the final questionnaire. Final adjustments After the reliability and validity assessment, four further alterations were made to the questionnaire. One amendment was made to question 1 in Section A, the tick box was removed from Other please specify (as the majority of participants omitted this question, despite earlier font and formatting alterations). The age range (question 23) was Table 2 Spearman s correlation coefficient for the final version of Section C Question N Test retest reliability a C7. Sugar in two servings C8. Grams of fat in half the pack C9. Servings in the product C10. Fat content in Product C11. Fat content in Product C12. Sugar content in Product C13. Comparison between Products 2 and C14. Lowest saturated fat content C15. Losing weight C16. Reducing salt intake a Significant at Po altered (from to 18 25) to remain consistent with the sample group used. Two additional questions were added to the questionnaire. Both questions were included in Section E of the questionnaire and assessed the participant s motivation to eat a healthy diet ( How motivated are you to eat a healthy diet?) and their own perceived knowledge of a healthy eating ( How would you rate your own knowledge of healthy eating on a scale of 1 10? (1 ¼ no knowledge and 10 ¼ very knowledgeable) ). The final version of the questionnaire (excluding Section E the demographic items) is shown in Appendix Table 1. Discussion The questionnaire development process involved a literature search, reviewing the findings from existing literature and highlighting any gaps in the current research. The review highlighted the limited availability of studies reporting the development of validated tools used to measure consumers food label reading habits. To our knowledge, this is the first study to report the validation and reliability testing of a short questionnaire to assess consumers use, understanding and perception of food labels.

5 214 Table 3 Item analysis for the final version of Section C Question Difficulty (% answering correctly) Discrimination (item-total r-value) Sugar content in two servings (Question 7) Grams of fat in half pack (Question 8) Servings in pack (Question 9) Fat content in Product 1 (Question 10) Fat content in Product 2 (Question 11) Sugar content in Product 3 (Question 12) Comparing the saturated fat content of two products (Question 13) Saturated fat content in product with lowest sat fat (Question 14) Losing weight (Question 15) Salt intake (Question 16) In the development of this instrument, special consideration was given to developing a questionnaire, which could be completed independently, without assistance. It was, therefore, important that the readability level of the questionnaire was pitched appropriately. Care was taken to avoid long sentences, complex terminology, acronyms or abbreviations, double-barrelled and leading questions. It is recommended that independently administered (self-completion) questionnaires are short in length and are composed mostly of closed structure questions (McColl et al., 2001). Postal questionnaires short in length (Edwards et al., 2002) and containing o1000 words (Jepson et al., 2005) are more likely to yield a high response rate. Care was taken to ensure that the questionnaire was short in length (only four sides of A4), contained o1000 words and comprised mainly closed end questions (47 out of 49 items). The response categories in the close structure questions were considered mutually exclusive and exhaustive after the literature review of the topic area and the considerable effort used during the face and content validity process. Various other formatting characteristics were consciously applied in the questionnaire construction. Using a booklet format with double-sided printing, vertical formats for closed structure responses, a font size of 10 points and a high text/background contrast were all incorporated into the instrument layout, in line with best practice recommendations for questionnaire design (McColl et al., 2001). Demographic questions were purposively inserted at the end of the questionnaire, as it has been suggested that these questions can be considered threatening (Sudman and Bradburn, 1982) or boring (Rattray and Jones, 2007). The demographic questions included in this questionnaire were adapted from existing tools, for example The 2001 Census, The Food Standards Agency Consumer Attitudes Survey and The Family and Children s Study. Although the questionnaire underwent vigorous reliability testing, limitations may still remain in the reliability methodology used. Repeat reliability testing (test retest) presumes that exposure to the questionnaire at time 1 does not trigger direct investigation of the topic area by the respondent. In this study, no intervention took place between survey contact; therefore, no changes would be expected unless enquiry of the subject area was prompted by the initial questionnaire exposure. The Spearman s correlation coefficients (0.51 to 0.97) were statistically significant at Po0.001, indicating strong reliability of these items assessed. Although the optimum value for Cronbach s alpha varies throughout the literature (DeVon et al., 2007), a value is the most commonly accepted figure (Bland and Altman, 1997). In this study, all values were indicating high levels of internal consistency in this instrument. The majority of the participants used in the reliability and validity testing of this questionnaire were both White and Scottish. This is recognized as a limitation of this study, as we cannot presume that all ethnic groups and nationalities would use nutrition labels in the same manner, nor interpret this questionnaire in the same way. Future feasibility testing in different geographic areas or with alternative ethnic groups should be conducted before using this tool as a guide to inform changes in policy. A further limitation of this study is the use of closed questions when assessing consumers attitudes to nutrition information in catering establishments. Closed questions are not considered suitable when investigating topics we have limited knowledge of (Rattray and Jones, 2007). Although accepting the possibility that information yielded from this instrument on nutrition information in catering establishments may not be representative of the entire population, it is notable that robust content and face validity were performed in the development of this questionnaire, and future qualitative studies in the topic area will expand on the findings derived from this instrument. The questionnaire developed in this study was intended to be administered by post, nationwide throughout the UK. Further research should involve examining other questionnaire administration methods in various settings. For example, administering the questionnaire in a supermarket or retail setting may provide valuable information when comparing the choices and decisions consumers make within their own home environment and in a busy retail or catering setting.

6 Given the short length of the questionnaire, questions included in the final draft were those considered fundamental to the topic area. Future research in this subject area may result in the expansion of this questionnaire to include more in-depth items, particularly on issues surrounding the catering industry. In conclusion, the final questionnaire is a standardized (self-complete) instrument for assessing consumers use, understanding and perception of food labels. Conflict of interest The authors declare no conflict of interest. Acknowledgements This research was funded by the Food Standards Agency Postgraduate Scholarship Scheme. We thank Dawn Mackison, Tommy Mackison and the staff at the Centre for Public Health Nutrition Research for their help with the packaging and dissemination of the questionnaire. References Black A, Rayner M (1992). Just Read the Label. The Coronary Prevention Group: London. Bland MJ, Altman DG (1997). Cronbach s Alpha. BMJ 314, Council Directive 90/496/EEC of 24 September on nutrition labelling for foodstuffs (1990). Council Directive 90/496/EEC of 24 September on nutrition labelling for foodstuffs. Council Directive 2000/13/EC of the European Parliament and of the Council of 20 March 2000 (2000). Council Directive 2000/13/EC of the European Parliament and of the Council of 20 March /13/EC. Cowburn G, Stockley L (2005). Consumer understanding and use of nutrition labelling: a systematic review. Public Health Nutr 8, Defra (2006). Family Food in National Statistics: London. DeVon HA, Block ME, Moyle-Wright P, Ernst DM, Hayden SJ, Lazzara DJ et al. (2007). A psychometric toolbox for testing validity and reliability. J Nurs Sch Second Quarter 39, Edwards P, Roberts I, Clarke M, DiGuiseppi C, Pratap S, Wentz R et al. (2002). Increasing response rates to postal questionnaires: systematic review. BMJ 324, EU (2008). Proposal for a Regulation of the European Parliament and of the Council on the Provision of Food Information to Consumers. Official Journal of the European Union: Brussels European Heart Network (2003). A systematic review of the research on consumer understanding of nutrition labelling. European Heart Network: Brussels. Flesch R (1948). A new readability yardstick. J Appl Psychol 32, Food Standards Agency (2006). Consumer Attitudes to Food Standards Food Standards Agency: London. Food Standards Agency (2007). Consumer Attitudes to Food Standards. Food Standards Agency: London. Grunert KG, Wills JM (2007). A review of European research on consumer response to nutrition information on food labels. J Public Health 15, Hawkes C (2004). Nutrition Labels and Health Claims: The Global Regulatory Environment. World Health Organisation: Geneva. Higginson CS, Rayner MJ, Draper S, Kirk TR (2002). The nutrition label which information is looked at? Nutr Food Sci 32, Institute of Grocery Distribution (1998). Voluntary Nutrition Labelling Guidelines to Benefit the Consumer. IGD. Jepson C, Asch DA, Hershey JC, Ubel PA (2005). In a mailed physician survey, questionnaire length had a threshold effect on response rate. J Clin Epidemiol 58, Kessler H, Wunderlich SM (1999). Relationship between use of food labels and nutrition knowledge of people with diabetes. Diabetes Educ 25, Kline P (1986). A Handbook of Test Construction. Introduction to Psychometric Design. Methuen & Co Ltd: London. Kline P (1993). The Handbook of Psychological Testing. Routledge: London. Krukowski RA, Harvey-Berino J, Kolodinsky J, Narsana RT, Desisto TP (2006). Consumers may not use or understand calorie labeling in restaurants. J Am Diet Assoc 106, Levy AS, Fein SB (1998). Consumers ability to perform tasks using nutrition labels. J Nutr Educ 30, Mannell A, Brevard P, Nayga RM, Combris P, Lee R, Gloeckner J (2006). French consumers use of nutrition labels. Nutr Food Sci 36, Marietta AB, Welshimer KJ, Anderson SL (1999). Knowledge, attitudes, and behaviors of college students regarding the 1990 Nutrition Labeling Education Act food labels. J Am Diet Assoc 99, McColl E, Jacoby A, Thomas L, Soutter J, Bamford C, Steen N et al. (2001). Design and use of questionnaires: a review of best practice applicable to surveys of health service staff and patients. Health Technol Assess 5, Microsoft (2007). Frequently asked questions about grammar proofing in Word. Microsoft Corporation. Mintel (2007). Eating Out Review Leisure Intelligence. Mintel International Group Limited: London. National Institute of Nutrition (1999). Nutrition labeling: perceptions and preferences of Canadians. Ottawa. Neuhouser ML, Kristal AR, Patterson RE (1999). Use of food nutrition labels is associated with lower fat intake. J Am Diet Assoc 99, New York City Department of Health and Mental Hygiene (2007). New York City Health Code (Repeal and reenact 81.50). Rattray J, Jones MC (2007). Essential elements of questionnaire design and development. J Clin Nurs 16, Rothman RL, Housam R, Weiss H, Davis D, Gregory R, Gebretsadik T et al. (2006). Patient understanding of food labels the role of literacy and numeracy. Am J Prev Med 31, Stuart SA, Schroder M, Hughes A, Bower J (2004). Dimensional analysis of schoolchildren s food label comprehension: a pilot study. Int J Cons Stud 28, Sudman S, Bradburn N (1982). Asking Questions. A Practical Guide to Questionnaire Design. Jossey-Bass Inc.: California. 215

7 216 Appendix Table 1 Food label questionnaire Please complete sections A, B, C, D ane E

8 217