FOOD FOR THOUGHT Topical Insights from our Subject Matter Experts THE NFL S SENSORY PANEL PHILOSOPHY

Transcription

1 FOOD FOR THOUGHT Topical Insights from our Subject Matter Experts THE NFL S SENSORY PANEL PHILOSOPHY The NFL White Paper Series Volume 3, June 2012

2 Overview Many descriptive panel styles exist and it can be confusing to try to understand how they differ and how those differences can influence the quality of the resulting data and business decisions. At The NFL, we believe that robust descriptive data can be collected in a variety of ways and we pride ourselves on choosing appropriate methods from our toolbox based on the needs of each individual project. One of the key ways that The NFL s panel approach is differentiated is that we conduct panel training for every project we run and this training is conducted by one or more sensory scientists with a Master s degree or Ph.D. in food science. Our panel leaders work both with our clients, to help design studies to best meet their project goals, and with our trained panelists, to focus them on the attributes of greatest interest for each specific project. We believe that extensive involvement from our experienced panel leaders is a key reason why we consistently receive client feedback about the high quality of our data. We also believe that it is beneficial to give our panelists some degree of freedom which allows them to focus on differences among products. When we consult with outside companies to help with descriptive training, we have found situations where panels show poor discrimination, likely because the panelists were given negative feedback for giving wrong scores or using the scale differently from each other. This has the unintended consequence of steering the panelists towards using safe scores that don t differentiate the products at hand. We encourage our panelists to use the scale to show differences among products and we provide non-threatening feedback to each panelist on both their ability to show differences among products and their consistency between replications. What is Descriptive Analysis? As the name implies, descriptive data describe the sensory properties of one or more samples, typically in a numeric manner. The questions we typically answer are What do the products taste (or smell) like? or How do the flavors (or textures) of the products differ? Examples are shown below. In Figure 1, we can see that all of the test yogurt products have less Total Strawberry Flavor, less Jammy Strawberry Flavor character, and more Candy Strawberry character than the reference product. Of the variants, sample #3 is the most different from the reference product in Strawberry flavor intensity and character. In Figure 2, the three breath mint products show different profiles over time. Sample A is most intense at early time points but decreases in Total Impact Intensity more quickly than Sample B. Figure 1: Example Spider Web Plot of Yogurts Figure 2: Example Plot of Total Flavor Intensity Over Time for Breath Mints Page 2 of 7

3 Consumers are good at telling us what they like or don t like, but they are typically not adept at clearly describing food flavor and texture nuances. In order to get actionable descriptive information, we use a team of trained panelists to collect our sensory profiles. The trained panelists are able to give us much more detailed and consistent information than untrained consumers can. Descriptive and consumer tests answer different types of research questions (for example, What do the products taste like? vs. Which product do consumers prefer? ), but they often compliment each other. When we have both consumer liking data and descriptive panel data on the same set of products, we can use the descriptive output to help understand the reasons behind consumer responses. The NFL s Descriptive Panelists The NFL has a team of approximately 40 highly trained panelists. They have all been trained to describe the appearances, aromas, flavors, and textures of a wide variety of food, beverage, and consumer products. We use panelists from our panelist pool for all of our descriptive project work. The current NFL panel has an average of 8 years of experience. This group has continually proven their ability to accurately describe the sensory character of products in great detail and to detect differences among products within studies. The panelists typically participate in numerous evaluations per week and work on a part-time basis. Client and product information is typically not shared with the panelists so that they can provide unbiased data. All of the panelists have signed a non-disclosure agreement to cover their work at The NFL. Panelist Selection Process Every few years, we replenish our panelist pool by recruiting and training a new group of approximately 15 panelists. Screening is typically conducted in three phases: phone screening of 150 applicants, on-site acuity testing of 100 individuals, and face-to-face interviews with advanced acuity screening for 30 individuals. Our goal is to select panelists with a high degree of food engagement that are adept at describing food aromas, flavors, and textures. Panel Training After a group of highly qualified panelists has been selected, they are put through several months of training before being integrated into our panelist pool to be used on client projects. During the training period, they expand their food sensory vocabularies, learn to use a 15-point point scale to rate attribute intensities, and evaluate a wide variety of food and personal care products. They are also exposed to several hundred reference standards that exemplify specific food sensory characteristics or intensities. For example, our sweetness intensity scale is anchored with several concentrations of sucrose in water and we illustrate sweet aromatic characteristics with references like vanilla extract or ethyl maltol (cotton candy character). The panelists-in-training hone their skills by participating in practice tests on many different types of products. After each test, they are given detailed feedback while retesting the products to help them improve their performance. Page 3 of 7

4 Ongoing Training and Performance Monitoring Training is not complete when the panelists enter our testing pool; we continually train our panelists with orientation sessions before every test we conduct. This enables them to be sensitive to the specific sensory notes that are important on each study and to continually hone their skills. Several times a month, the panelists are given visual performance feedback to help them maintain their calibration. A panel leader will retaste the products with the panelists as they review their scores to highlight potential areas for improvement. Feedback is given both on discrimination among products and consistency between replications. Descriptive Methodologies We employ several core descriptive methodologies, including Quantitative Descriptive Analysis, Narrative Descriptive Analysis, and Consensus Descriptive Analysis. We choose from these tools based on the needs of each individual project. Quantitative Descriptive Analysis This is our most widely used descriptive method. We typically use 10 of our trained panelists for our quantitative panel work. The panelists rate product appearance, aroma, flavor, and texture attribute intensities on 15-point line scales. Orientation Session We begin almost every quantitative project with one or more 2-hour orientation sessions. During these sessions, the panel leader works with the panelists to discuss evaluation methods, review the products, develop a ballot, and define each attribute. In addition, reference standards that exemplify key attribute characters and intensities are reviewed to make sure that the panelists language and scale usage are aligned. An example of an abbreviated yogurt flavor ballot with references listed on the line scales is shown in Figure 3. Figure 3: Example Quantitative Descriptive Ballot for Strawberry Yogurt Note: Typical projects would also include aroma, texture, and appearance attributes. Page 4 of 7

5 Data Collection Quantitative data collection is conducted in sensory testing booths that are partitioned so that the panelists are separated from each other. Products labeled with three-digit numbers are served one at a time and the panelists rate the intensity of each sensory attribute on the computerized ballot by clicking on the line-scales. Presentation order is randomized such that each product is seen approximately an equal number of times in each possible position. Two data replications are collected from 10 panelists for most projects. Between samples, the panelists clear their mouths by rinsing with water, eating unsalted soda crackers, and using other palate cleansers as needed. Although testing rate is highly dependent on the nature of the products, up to eight samples can usually be evaluated in a 2-hour session for a typical product that isn t overly fatiguing. Statistical Analysis The panelists line scale responses are converted to numbers ranging from 0 to 15 within SIMS (Sensory Information Management System). Mean intensities are then calculated for each sensory characteristic. Analysis of Variance and multiple comparison tests (such as Duncan s Multiple Range Test) are used to determine if the differences among the samples are statistically significant for each attribute. An example of an abbreviated means table is shown in Figure 4. Graphs, such as Figures 1 and 2, are also created to help visualize differences among the products. When appropriate, the NFL s statistical team will also use multivariate modeling techniques to interpret the data. Figure 4: Example Descriptive Means Table for Strawberry Yogurt Note: Typical projects would also include aroma, texture, and appearance attributes. Reference Yogurt Yogurt #1 Yogurt #2 Yogurt #3 P-Value Confidence Level FLAVOR: b a ab c Total Flavor ** a c b d Total Strawberry < ** a c b d Jammy Strawberry < ** c b b a Candy Strawberry < ** c a bc b Cultured Dairy < ** a b ab a Sweet Aromatic * a b a c Sweet ** b a b a Sour < ** AFTERTASTE: Aftertaste NSD n = 20 (10 panelists, 2 evaluations each) Analysis of Variance Confidence Levels: *=90%, **=95% NSD: Not significantly different at confidence levels of 90% or higher. Page 5 of 7

6 Narrative Descriptive Evaluations Narrative Descriptive Analysis is used when our clients need in-depth written descriptions of product sensory characteristics but do not need quantitative data or statistical analysis. This method is often used when clients need rapid feedback comparing their product to another brand or comparing a current formulation to a modified formulation. An NFL panel leader and three members of The NFL s trained descriptive panel typically participate in these evaluations. The panelists independently evaluate each sample and describe the major appearance, aroma, flavor, and texture characteristics of the reference sample. They then indicate the specific sensory attributes that differ in the variant samples and to what degree the variant samples differ from the reference sample in the specific attributes noted. In addition, they will make note if a variant sample shows consumer relevant downsides compared to the reference sample. All of the information collected is based upon a consensus of opinion reached though group discussions under the guidance of the panel leader. We have found the Narrative Descriptive output to be easily understood by all audiences; an example of a typical output is shown in Figure 5. Figure 5: Example Narrative Descriptive Output for Strawberry Yogurt Color/Apperance Aroma Reference Product Bright pale pink, thick, small darker pink specks, some small red pieces of strawberry Moderate strawberry, slight to moderate sweet aromatic, slight cultured dairy Test Product - Deviation from Reference Slightly darker pink, slightly thinner Moderately less strawberry slightly less sweet aromatic Flavor Moderate strawberry (fresh fruit character), moderately sweet, slightly sour, slight cultured dairy Texture Moderately thick, smooth Slightly thinner Moderately less sweet, slightly more cultured dairy, slightly less strawberry, slightly different strawberry character (more candy-like) Difference from Reference: Slightly to Moderately Different from the Reference Product Consumer Relevance: Slight Downside Conclusions: The sample has moderately less strawberry aroma, is moderately less sweet, and has a slightly different strawberry flavor character (more candy-like) than the reference product. Consensus Descriptive Analysis During consensus descriptive evaluations, three panelists individually rate each sample for each sensory attribute of interest. A panel leader then verbally collects the scores and a consensus value is reached for each attributed based on a group discussion. The output looks similar to a quantitative descriptive means table except that there are no statistical comparisons. At The NFL, we feel that there are some occasions, such as shelf-life evaluations, where rapid quantitative output is needed. However, we feel that quantitative descriptive analysis gives much more robust data and we strongly caution against using consensus scores for multivariate modeling. In cases where cost or rapid timing are concerns, we find that our narrative descriptive offering provides high quality and actionable information. Page 6 of 7

7 The NFL s Sensory Facility The NFL s 39 sensory booths serve as controlled environments for descriptive research. Partitions between each booth ensure that the panelists provide independent responses. The booths are equipped with computerized data entry software and have controlled lighting and ventilation in accordance with standard sensory evaluation practices. Each set of booths is serviced by its own preparation kitchen to ensure optimal efficiency and control. The four sensory kitchen areas provide ample space for complex preparation of a wide range of products. They are equipped with a variety of appliances including stoves, microwaves, freezers, and refrigerators. Orientation sessions and consensus panels take place in The NFL s panel discussion rooms. These two rooms have round tables to facilitate discussions and have controlled lighting and ventilation. Our focus group room is also used for descriptive orientation sessions when clients are interested in watching the trained panelists at work. The NFL s Sensory Panel Leaders Sharon McEvoy leads The NFL s Sensory Evaluation division and has over 25 years of experience in the field. She received her B.S. in Food Science from Penn State and her Master s from UC Davis, studying under Rose Marie Pangborn. She began her sensory career with Pepsi-Cola, working in their R&D center for 7 years. Dawn Chapman runs descriptive projects in addition to managing The NFL s Sensory Statistics group and has 10 years of sensory experience. She studied mathematics and chemistry at DePauw University and received a Ph.D. in food science with a sensory emphasis from UC Davis. Regan Javier leads descriptive panels and is the key contact for sensory shelf-life testing at The NFL. She received her Master s Degree from UC Davis and has 6 years of experience in the field. For more information please contact: Sharon McEvoy at McEvoyS@TheNFL.com, Dawn Chapman at ChapmanD@TheNFL.com, Page 7 of 7