AN ALTERNATIVE METHOD FOR ASSESSING LIKING: POSITIONAL RELATIVE RATING VERSUS THE 9-POINT HEDONIC SCALE ABSTRACT

Transcription

1 AN ALTERNATIVE METHOD FOR ASSESSING LIKING: POSITIONAL RELATIVE RATING VERSUS THE 9-POINT HEDONIC SCALE S.M. CORDONNIER and J.F. DELWICHE 1 Department of Food Science and Technology The Ohio State University 2015 Fyffe Road Columbus, OH Accepted for Publication June 3, 2007 ABSTRACT An alternative rating method, known as positional relative rating (PRR), or rank rating, was compared with the traditional 9-point hedonic scale. Initially, category scale and line scale versions of PRR were investigated to see which of either was better at differentiating samples. Four lemonades were assessed in duplicate on both scales (category and line). Overall quality, sweetness intensity and sourness intensity were assessed in separate sessions by groups of ~50 consumers, depending on session. The category scale better differentiated samples than did the line scale for two of three attributes assessed, with differentiation being the same in the third. Subsequently, the overall liking of these same lemonades was assessed by 100 consumers on both the PRR category scale and the traditional 9-point hedonic scale. The differentiation of samples for both scales was equivalent, although mean values were somewhat lower with PRR. PRACTICAL APPLICATIONS This work explores an alternative to the traditional 9-point scale. Positional relative rating (PRR) performs as well as the traditional 9-point scale in differentiating samples, suggesting that it is a viable alternative. However, as PRR seems to give consistently lower mean values than does the traditional 9-point hedonic scale, those using a hurdle approach to screen products may wish to lower that hurdle when PRR is used. Findings also suggest that consistency errors are reduced when samples are presented simultaneously 1 Corresponding author. TEL: ; FAX: ; delwiche.1@osu.edu 284 Journal of Sensory Studies 23 (2008) All Rights Reserved. 2008, The Author(s) Journal compilation 2008, Blackwell Publishing

2 ALTERNATIVE FOR ASSESSING LIKING 285 (rather than in a serial monadic fashion) and judges are allowed to adjust their ratings. INTRODUCTION The 9-point hedonic scale, originally developed by Peryam and Girardot (1952), is the most commonly used scale for assessing liking and preferences of foods. It is a balanced scale around a neutral point with category labels that have been determined by Jones and Thurstone (1955) to be fairly evenly spaced psychologically. The adjectives used to label the nine points also aid in interpretation of panel means. Typically, the hedonic scale is used with untrained panelists who frequently use the product of interest, and it is considered the gold standard for assessing degree of liking. Despite the widespread acceptance of this practice, a superior alternative to it (and all other scales and sensory methods) is constantly being sought. In most cases, when the 9-point hedonic scale is used, consumers are presented with one sample at a time and are asked to make their rating without reference to the prior or future samples. Generally, panelists only see the current scale and are not allowed to change their prior responses. Sometimes, not only are assessments made without reference to other samples, but the products themselves are presented in complete isolation, with one sample being removed from the booth before the next one is presented. Assessing samples sequentially without reference to others in the set is known as a serial monadic approach. Such an approach presupposes the hidden assumption that ratings are made upon an absolute scale, and that when panelists assess a product, they are comparing it to a set of internalized standards (Mellers 1983; Koo and Kim 2002). A serial monadic presentation style is adopted so that the presence of one sample does not interfere with the perception of and judgment on other samples. Alternatively, one could present all samples at the same time and allow individuals to not only taste samples once, but also to retaste them, refer back to prior ratings and adjust those ratings until satisfied with the assessments of all samples. This approach assumes that ratings are made upon a relative scale, i.e., when a panelist assesses a product, the rating of one product is relative to the ratings of the other products being considered (Mellers 1983; Koo and Kim 2002). The further apart two sample ratings are, the more different they are from one another. If, in fact, relative assessments are being made, simultaneous presentation would allow the panelist to more accurately rate these distances between products because the memory of stimuli could be refreshed whenever necessary by retasting (Lee and Kim 2001). Positional relative rating (PRR), also known as rank rating (e.g., Kim and O Mahony 1998; Lee and Kim 2001; Koo and Kim 2002; O Mahony et al.

3 286 S.M. CORDONNIER and J.F. DELWICHE 2004), relies upon the panelist comparing all the samples with one another. With PRR, the panelist receives all samples at once. As each sample is tasted, the sample is physically placed along a representation of the scale on a paper strip. With each assessment, the panelist is allowed to adjust the position (and associated rating) of the samples, until the panelist is satisfied with all assessments. Retasting the samples is allowed, and is even encouraged by the nature of the protocol. Research has shown that with PRR, panelists are less prone to making reversal errors, i.e., rating a lower concentration as more intense than the higher concentration (Kim and O Mahony 1998; Park et al. 2003). Thus, it seems plausible that if PRR was used to assess consumer liking of products, they might make less errors than when using the 9-point hedonic scale. Recent work by Ishii et al. (2007) lends support to this premise. They showed that untrained judges performing simple descriptive analysis reach a criterion level of performance more quickly when panelists make assessments using an attribute-by-attribute protocol (which encourages relative scaling) than when they make assessments using a serial monadic protocol (which encourages absolute scaling). Thus, the primary goal of this research was to compare liking assessments made with the traditional 9-point hedonic scale with those made with PRR. Its potential in hedonic judgments has not yet been fully explored, although initial work has been conducted in Korea (O Mahony et al. 2004). Before PRR and 9-point hedonic scale assessments were directly compared, category and line scale versions of PRR were both investigated to see which if either was better at differentiating samples. EXPERIMENT 1 Before the traditional 9-point was compared with PRR, additional investigation of PRR was conducted to determine if it was more discriminating with a semistructured continuous line scale or as a category scale. To this end, four unique formulations of lemonade were used as test stimuli and the subjects assessed these stimuli on both scales. Methods Materials. Test stimuli were four different formulations of lemonade made from Kool-Aid (Kraft Foods, Rye Brook, NY) concentrated powder. Standard lemonade was made following package directions with 3.4 g/l of concentrated powder and 100 g/l of sucrose (Sigma sucrose minimum 99.5%, Sigma-Aldrich Inc., St. Louis, MO). Sweetened lemonade contained 1.5 times the sucrose concentration (150 g/l) of the package recipe. Sour lemonade was

4 ALTERNATIVE FOR ASSESSING LIKING 287 made by adding 5.6 g/l of citric acid (Fisher Chemicals, Fair Lawn, NJ) to the package recipe. Finally, high-impact lemonade was made by adding both sucrose and citric acid (150.0 and 5.6 g/l, respectively) to the package recipe. All lemonades were made with reverse osmosis water (Millipore, RiOs16, Millipore Corporation, Bedford, MA). All stimuli were presented in 1-oz translucent plastic cups (Solo Plastic Soufflés, Solo Cup Company, Baltimore, MA). Subjects. Volunteer participants were recruited from the lobby area of Parker Food Science Building at the Ohio State University (OSU). Three sessions were conducted on separate days. For the first session, the panel consisted of 23 men and 28 women (ages 18 55). For the second session, the panel consisted of 15 men and 34 women (ages 18 65). For the third session, the panel consisted of 23 men and 28 women (ages 18 over 65). Procedures. All procedures were approved by the OSU Office of Responsible Research Practices before experimentation began. For each session, volunteer consumers evaluated each sample in a complete block design with sample order counterbalanced across panelists. The first panel assessed overall quality, the second panel sourness intensity and the third panel sweetness intensity. In each session, half of the panelists started with the line scale, and half started with the category scale. In all sessions, evaluations were made under normal illumination (i.e., red lights were not used). All assessments were made using PRR. In these assessments, the participant tasted all eight samples (two presentations of each of the four lemonades) and placed each along a paper strip ( in. long) marked with a scale, such that the position on the scale represented the perceived intensity of the attribute being assessed. The structured line scale was 30-in. long, marked at 3.25-in. intervals with 0.10-in. tick mark numbered from 1 (low) to 9 (high). The categorical scale was also 30 in. long, with 17 boxes ( in.) positioned at 1.75-in. intervals; every other box was numbered 1 9, as with the line scale. Consumers rated each sample in duplicate on both scales, testing one scale at a time. They were allowed to retaste the samples and to adjust their responses before transferring the final responses onto a scaled-down version of the same scale on a computer screen. Data were collected with Compusense five version 4.6 (Compusense Inc., Guelph, Ontario, Canada). All eight samples were presented simultaneously at refrigeration temperature, which ranged from 12.5 to 15.5C. In order to minimize fatigue, after completing assessments of all eight stimuli on one scale type, the panelists were asked to rinse for 40 s before continuing their assessments on the alternative scale type. Room temperature bottled spring water (Ice Mountain Water Co., Hilliard, OH) was provided for rinsing. The same process was repeated

5 288 S.M. CORDONNIER and J.F. DELWICHE for each session (overall quality, sweetness and sourness). Although some individuals participated in more than one panel, their performance over sessions was not tracked. Statistical Analysis Data were analyzed with repeated measures analysis of variance (ANOVA) and Scheffé s post hoc analysis where appropriate using Statistica (version 7.0, StatSoft, Inc., Tulsa, OK). In addition, power was calculated for each scale in all three sessions using the calculation of Keppel and Wickens (2004). Results and Discussion As shown in Table 1, when sweet intensity was assessed with the category scale, the sweet lemonade was rated as significantly sweeter than all other samples (Scheffé s, P < 0.05) while the sour lemonade was rated as significantly less sweet than all the other samples (Scheffé s, P < 0.05). In contrast, when sweet intensity was assessed with the line scale, sour lemonade could not be differentiated from standard lemonade (Scheffé s, P > 0.05), nor could sweetened lemonade be distinguished from the high-impact lemonade (Scheffé s, P > 0.05). Similarly, when overall quality was assessed, there was more discrimination of the lemonades with the category scale than with the line scale (Table 1). With the category scale, high-impact lemonade was rated significantly higher than all the other stimuli (Scheffé s, P < 0.05). Additionally, standard and sweetened lemonades were significantly different from each other (Scheffé s, P = 0.047). In contrast, assessments made with the line scale TABLE 1. MEAN ATTRIBUTE RATINGS OF EACH LEMONADE FOR CATEGORY AND LINE SCALES Attribute Lemonade formulation ANOVA P value Sour Standard Sweet High impact Category scale Sweet 3.7 a 5.0 b 6.2 c 5.3 bc < Sour 6.5 b 2.9 a 2.6 a 6.1 b < Quality 4.1 ab 3.3 a 4.3 b 5.6 c < Line scale Sweet 4.0 a 4.9 a 6.5 b 5.3 b < Sour 6.3 b 2.8 a 2.8 a 5.8 b < Quality 4.2 a 3.6 a 4.5 a 5.8 b < Within a row, means with the same superscript letters are not significantly different from one another (Scheffé s, P > 0.05). ANOVA, analysis of variance.

6 ALTERNATIVE FOR ASSESSING LIKING 289 found no significant differences between standard and sweetened lemonades (Scheffé s, P > 0.05). In contrast, when sour intensity was assessed, the category and line scales showed the same discrimination of the lemonades (Table 1). Both scales found significant differences (Scheffé s, P < 0.05) only between the lemonades with added citric acid (sour and high-impact lemonades) and those without (standard and sweet lemonades). Power for each scale in all three sessions was >0.90. Despite this, the category scale was equally or more effective at differentiating samples than the line scale for all three attributes. Thus, the category scale was used for the comparison of PRR with the traditional 9-point scale. EXPERIMENT 2 Using a single group of subjects, assessments of overall liking on the traditional 9-point hedonic scale was compared with the alternative PRR category scale. Methods Materials. All materials used were identical to those used in Experiment 1. Subjects. As before, volunteers were recruited from the Parker Food Science Lobby area. Participants were 50 men and 50 women ranging in age from 18 to 65. Procedure. All procedures were approved by the OSU Office of Responsible Research Practices before experimentation began. Assessments with the PRR category scale followed the same protocol as was used in Experiment 1. Although tasting of samples was the same for the 9-point hedonic scale, in this condition, the scale appeared only on the computer screen, without a paper scale mirroring what was found on the screen on the booth counter. However, so as not to confound the scale with the presentation method, the participants were still allowed to retaste the samples and to change their answers before submitting; all samples were presented on the screen simultaneously. As before, all eight samples (two presentations of each lemonade formulation) were presented simultaneously. The samples were served at refrigeration temperature, which ranged from 12.5 to 15.0C. After completing assessments of all eight stimuli on one scale type, the panelists were asked to rinse for 40 s before continuing their assessments on the alternative scale type

7 290 S.M. CORDONNIER and J.F. DELWICHE in order to minimize fatigue. Half of the panelists began with the 9-point hedonic scale, while the other half began with PRR. Room temperature bottled spring water (Ice Mountain Water Co.) was provided for rinsing, and testing was conducted under white light. Statistical Analysis Data were subjected to repeated measures ANOVA and Scheffé s post hoc analysis where appropriate using Statistica (version 7.0, StatSoft, Inc.). In addition, power was calculated for each scale using the calculation of Keppel and Wickens (2004). Results and Discussion As shown in Table 2, the traditional 9-point hedonic scale and the PRR category scale show the same discrimination for all samples (Scheffé s, P < 0.05). All four lemonade samples were rated roughly 0.4 points lower with PRR, which would have important ramifications in an environment where product decisions are based upon a hurdle. However, additional research is needed to determine if this is a consistent difference between the two protocols or if it was unique to this investigation. Because liking was rated rather than intensity, there was no physical dimension underlying this psychological assessment, making it impossible to assess the extent of reversals with either scale. As an alternative, individual responses were examined for consistency errors. If the rank order of samples placed each replicate next to its pair, then no consistency error was apparent. If, however, the rank order of samples placed another sample(s) between the replicate pairs, this was considered to be a consistency error. PRR and the 9-point hedonic scale showed little difference in consistency errors, with 114 being found with PRR and 107 being found with the 9-point hedonic scale. TABLE 2. MEAN LIKING RATINGS OF EACH LEMONADE FOR THE 9-POINT HEDONIC SCALE AND PRR Lemonade formulation ANOVA P value Standard Sweet Sour High impact 9-point hedonic 4.7 a 5.3 ab 4.7 a 5.9 b < PRR 4.3 a 4.8 ab 4.3 a 5.5 b < Within a row, means with the same superscript letters are not significantly different from one another (Scheffé s, P > 0.05). PRR, positional relative rating; ANOVA, analysis of variance.

8 ALTERNATIVE FOR ASSESSING LIKING 291 Similarly, the calculated power associated with PRR and with the hedonic scale were equivalent, both being >0.90. Thus, PRR and the 9-point hedonic scale seem to be equivalent in differentiation of samples, consistency errors and statistical power. GENERAL DISCUSSION As mentioned earlier, O Mahony et al. (2004) did some preliminary work that compared PRR with the traditional 9-point hedonic scale. They had 100 Korean consumers assess their liking of seven different toothbrushes and of seven different potato chips under three different conditions. First, stimuli were presented one at a time; ratings were made on a Korean translation of the traditional 9-point hedonic scale, and each stimulus was removed from the booth before the next stimulus was presented. In the two remaining conditions, the PRR of the samples was conducted using either the traditional 9-point structured hedonic scale or a 21-point unstructured hedonic scale. In both instances, all samples were presented simultaneously, and the judges were allowed to retaste the samples and to alter their assessments until they were satisfied. For the toothbrushes, all three conditions resulted in the same differentiation of the samples, identifying five significantly different groups which were in the same rank order for all three conditions (O Mahony et al. 2004). However, for potato chips, they did see some difference in differentiation across the conditions, as well as differences in rank order. The first condition differentiated only four groups of samples, while the other two conditions differentiated five. It is difficult to make direct comparisons between the findings of O Mahony et al. (2004) and the current study as there were several key ways in which the studies differed. The subjects were from different cultures, and thus the 9-point hedonic scale was presented in the native language of the judges, Korean for O Mahony et al. (2004) and English for the current study. In addition, the studies differed in how the samples were presented when assessments were made on the traditional 9-point scale: O Mahony et al. (2004) used a strict serial monadic protocol, while all stimuli were presented simultaneously in the current investigation. The use of the serial monadic approach made it impossible for O Mahony et al. (2004) to determine if differences between the conditions were due to the use of the PRR protocol of placing the samples along the scale or instead be due to the opportunity wherein the judges had to retaste the stimuli and alter their responses. The format of the current study removed this ambiguity by allowing the judges in the traditional 9-point scale condition to retaste the samples and to

9 292 S.M. CORDONNIER and J.F. DELWICHE alter their responses as they desired, ensuring that in both conditions, the judges could use the same cognitive strategy. Under these circumstances, the results of the 9-point hedonic scale and the PRR were strikingly similar, suggesting that what is key to reducing consistency errors is a procedure that allows for side-by-side comparisons and adjustment of ratings. REFERENCES ISHII, R., CHANG, H.-K. and O MAHONY, M A comparison of serial monadic and attribute-by-attribute protocols for simple descriptive analysis with untrained judges. Food Qual. Prefer. 18, JONES, L.V. and THURSTONE, L.L The psychophysics of semantics: An experimental investigation. J. Appl. Psychol. 39, KEPPEL, G. and WICKENS, T.D Design and Analysis: A Researcher s Handbook, Pearson Prentice Hall, Upper Saddle River, NJ. KIM, K.-O. and O MAHONY, M A new approach to category scales of intensity I: Traditional versus rank-rating. J. Sensory Studies 13, KOO, T.-K. and KIM, K.-O Effects of forgetting on performance on various intensity scaling protocols: Magnitude estimation and labeled magnitude scale (green scale). J. Sensory Studies 17, LEE, H.-J. and KIM, K.-O Effects of forgetting on various protocols for category and line scales of intensity. J. Sensory Studies 16, MELLERS, B.A Evidence against absolute scaling. Percept. Psychophys. 33, O MAHONY, M., PARK, H., PARK, J.Y. and KIM, K.-O Comparison of the statistical analysis of hedonic data using analysis of variance and multiple comparisons versus an R-index analysis of the ranked data. J. Sensory Studies 19, PARK, J.Y., JEON, S.Y., O MAHONY, M. and KIM, K.-O Induction of scaling errors. J. Sensory Studies 19, PERYAM, D. and GIRARDOT, N.F Advanced taste-test method. Food Eng. 24, 58 61, 194.