Is Web Eye-Tracking Good Enough?

Transcription

1 Is Web Eye-Tracking Good Enough? Recently, several agencies have emerged offering Web Eye-Tracking (i.e. eye-tracking via webcams) as a lower-cost substitute for traditional eye-tracking studies (which are conducted in-person). PRS IN VIVO pioneered the use of eye-tracking in consumer research over 40 years ago and today, we use (in-person) eye-tracking in several hundred packaging, shopper and new product studies annually. Thus, we were eager to gauge the viability of Web Eye- Tracking, to determine if we should offer this service to our clients. To this end, we conducted a series of six (6) pilot studies with two leading suppliers. These quantitative studies involved over 2,200 interviews across three different countries and in most cases, they were conducted in parallel to in-person eye-tracking studies, allowing us to directly compare findings. While these pilot studies suggest that Web Eye-Tracking has potential value, they also identified several significant limitations tied to both study execution and accuracy. Executing the Studies In order to track eye movement, Web Eye-Tracking services require study participants to allow access to their webcams and/or to download and install software on their computers. Then, they must go through a calibration process before starting the eye-tracking process. In our experience (with both providers), these proved to be significant barriers, as: About 2/3 of study participants (who took the survey) declined/refused to enter the eye-tracking process. 500 Frank W. Burr Blvd., Suite 5, Teaneck, NJ P:

2 2 Among those who did so, less than 50% went through calibration and were successfully tracked via their Webcams. Thus, the success rates were roughly 10%. In one of the largest pilot studies, there were 53 usable responses from a sample of 500 shoppers. In several other pilot studies, the number of usable responses was between people, resulting in Qualitative findings. While these executional issues are potentially surmountable, they do draw into question the cost/time-efficiency and project-ability of Web Eye-Tracking at this time. In particular, clients should be sure to confirm sample sizes (of completed interviews) in Web Eye-Tracking studies. Accuracy of the Eye-Tracking Data Of course, a second major challenge is the accuracy and validity of the data gathered from Web Eye-Tracking. Here, the providers openly acknowledge that there is a gap relative to in-person eye-tracking. To quote one executive: Our precision is somewhat lower than conventional (inperson) eye-trackers, so we are not always comfortable with analyzing very small areas. In fact, this quality gap can be quantified on a technical level, as in-person eye-tracking typically generates accuracy within 0.5 degrees of visual angle, while Webcam-based systems produce accuracy within 2-5 degrees. To put this in perspective, an error range of 2-5 degrees translates to a range of about 2.7 inches (7 cm) on a computer monitor, if a person is seated 22 inches away (56 cm) from the screen. And notably, a typical pack might only measure a few inches in each direction. In addition, an in-person eye-tracker records eye fixations at frames per second, while Webcams typically record between 5-30 frames per second. And in our considerable experience, we ve found it nearly impossible to accurately analyze fixations at a rate lower than 15 frames per second, as the human eye moves far faster than this. Yet while the quality gap is clear, the most relevant question (to most clients) is whether Web Eye-Tracking is good enough in terms of providing accurate data (and presumably leading to valid conclusions and informed business decisions). To answer this question, we focused on parallel studies, in which we tracked the same shelves and packs, using both in-person and Web-based methods:

3 3 Eye-Tracking of Shelf Sets (Visibility) At PRS IN VIVO, we regularly apply eye-tracking (conducted in-person, with life size shelf sets) to gauge the visibility of new products, packaging or shelving systems. Visibility (% Noting) is measured quantitatively, via monadic studies with base sizes of per cell and it is often an Action Standard within our studies, due to its proven link to in-market sales. Unfortunately, Web Eye-Tracking proved to be quite disappointing in gauging shelf visibility. As this example shows, % Noting figures were consistently higher in Web ET studies (vs. inperson), although not in a consistent manner that might allow for a simple correction factor. For some brands, reported visibility levels were 20%-30% higher via the Web Eye-Tracking, while the gaps were far smaller (5-10%) for other brands. These artificially high figures could lead to misguided assumptions and decisions. % Noting In-Person Web Brand #1 43% 41% Brand #2 55% 67% Brand #3 53% 78% Brand #4 18% 24% Brand #5 51% 89% Brand #6 64% 78% Brand #7 73% 100% Brand #8 41% 63% Brand #9 19% 33% Brand #10 55% 76% Brand #11 35% 63% Our leading hypothesis is that data gaps are a function of showing a shelf (of products) on a computer monitor. The different packs obviously appear quite small and the wider error range of Web Eye-Tracking (nearly 2-3 inches) often leads to misidentifications. In fact, to mitigate this issue, some companies have taken to showing competitive sets (rather than actual shelf sets) - and/or creating artificially wide gaps between brands on shelf - when using Web Eye-Tracking. However, this approach is self-defeating, as the artificial context makes it impossible to generate an accurate sense of a brand s shelf visibility in a real shelf (regardless of the accuracy of the eye-tracking).

4 4 Eye-Tracking of Individual Packages (Viewing Patterns) In our studies, pack viewing patterns are used primarily as a diagnostic measure, to help uncover the source of issues and guide refinements. For example, if shoppers are not playing back a particular product benefit, pack viewing may reveal/confirm that a related claim is recessive (and frequently overlooked when shoppers first view the packaging). While there are concerns tied to study execution and sample sizes, Web Eye-Tracking shows some promise in tracking viewing patterns for individual packs. As the example shows, % Noting figures (for individual elements on packaging) gathered via Web Eye-Tracking were generally consistent with those gathered in person. While there were significant differences on some elements (particularly on Cracker Jack branding and the boy visual), attention to claims (such as Prize Inside ) was similar across approaches. In this case (and others we reviewed), the research would be likely to drive similar conclusions regarding: The starting point and primary viewing flow Which messages were consistently seen (and missed) Notably, we found Web Eye-Tracking of pack viewing patterns to be more accurate than the VAS algorithm, which we reviewed previously and cannot recommend for any applications.

5 5 Why does Web Eye-Tracking appear to be more accurate for individual packs (vs. shelves)? Our leading hypothesis is that individual packs appear at nearly life size on computer monitors and thus can accommodate a wider error range (than small packs within a shelf set). Thus, Web Eye-Tracking is more likely to be good enough (to provide accurate direction) within this context. However, the size of image issue does suggest significant limitations, if Web Eye- Tracking is to be used on smaller screens and/or mobile devices or for tracking viewing of small and copy- intensive packs. Conclusions & Research Implications Based on our parallel research, PRS IN VIVO does not recommend using Web Eye-Tracking for measuring shelf visibility. Due to its accuracy limitations and the small size of packs on computer screens Web Eye-Tracking is too likely to provide misleading information, leading to misguided decisions. However, Web Eye-Tracking may hold promise in documenting how shoppers view individual packages, if current challenges (tied to study execution and sample sizes) can be overcome. Thus, our experience suggests that researchers should regard Web Eye-Tracking (of individual package viewing patterns) as a potential diagnostic for online surveys, in cases when an in-person eye-tracking study is not feasible. For more information, please contact Emilie Eberly at PRS IN VIVO at Emilie.Eberly@prs-invivo.com.