The Role of Visual Attention in the Managerial Judgment of Balanced-Scorecard Performance Evaluation: Insights from Using an Eye-Tracking Device

Transcription

1 DOI: / X Journal of Accounting Research Vol. 54 No. 1 March 2016 Printed in U.S.A. The Role of Visual Attention in the Managerial Judgment of Balanced-Scorecard Performance Evaluation: Insights from Using an Eye-Tracking Device YASHENG CHEN, JOHNNY JERMIAS, AND TOTA PANGGABEAN Received 4 February 2014; accepted 14 November 2015 ABSTRACT This paper investigates the role of visual attention in managerial judgments during balanced-scorecard performance evaluations. Using the Locarna eye tracker to establish the amount of time managers spent focused on School of Management, Xiamen University; Beedie School of Business, Simon Fraser University and JCU Singapore; College of Business Administration, California State University, Sacramento. Accepted by Christian Leuz. The authors thank the Social Sciences and Humanities Research Council of Canada (SSHRC) for financial support. This project was funded by an SSHRC Research and Development Initiatives Grant. We thank Rajiv Banker, Hsihui Chang, and Mina Pizzini for sharing their research instrument with us. This paper has benefited from comments and suggestions of the Editor and the two anonymous reviewers of this journal, Michael Shields, Steven Salterio, Peter Clarkson, Theresa Libby, Peter Tingling, Dennis Chung, Craig Emby, Irene Gordon, and Kim Trottier as well as seminar participants at Simon Fraser University, Nanjing University, Peking University HSBC Business School, the 2012 American Accounting Association Annual Conference, and 2013 European Accounting Association Annual Congress. The research instrument is available upon request from the corresponding author. An Online Appendix to this paper can be downloaded at Copyright C, University of Chicago on behalf of the Accounting Research Center, 2015

2 114 Y. CHEN, J. JERMIAS, AND T. PANGGABEAN visual cues, we found that managers who look more at strategically linked performance measures are more likely to make decisions consistent with the achievement of their subordinates strategic objectives. When aware of strategy, managers focused more on strategically linked performance measures than on nonlinked measures. The presentation format of the strategy information did not significantly affect this focus. Our findings indicate that awareness of strategically linked performance measures, but not their presentation, appears to be important in helping managers to make better decisions. This study contributes to the management accounting literature by generating useful insights into the impact of visual attention on judgments and decision-making processes. JEL codes: C91; D23; D83; M12; M41 Keywords: visual attention; balanced scorecard; eye tracking; managerial judgment 1. Introduction This study examines the role of visual attention in the managerial judgments made during balanced-scorecard (BSC) performance evaluations. Examining the relationship between attention and judgment is crucial to understanding managers behavior (Ocasio [1997]). Managers limited capacity to process information requires them to be selective about the types of information they use (Simon [1947]). Hoffman and Ocasio [2001] proposed that attention involves the subjective interpretation of available information, which requires managers to ignore some aspects of the information to process others more effectively. Lavie et al. [2004] found that attention guides the interpretation of information and, in turn, influences behavior. Previous studies in accounting have emphasized the importance of attention in understanding managers behavior. Kaplan and Norton [1996], for example, asserted that the BSC is designed to focus top executives attention on companies critical performance indicators. Lipe and Salterio [2000] stated that managers tended to pay insufficient attention to leading indicators and nonfinancial measures, which defeats the purpose of implementing a BSC. Although the importance of attention has been recognized, scant research in accounting has measured it directly or investigated its effect on managerial judgment. Previous studies in accounting assume that managers pay attention to any cue that can reasonably be predicted and researchers determine ex ante significant relationships between those cues and decision variables, without directly measuring the level of attention (Birnberg and Shields [1984]). Previous studies have been unable to measure levels of attention because of the lack of devices or software programs that effectively track the cues to which managers respond when processing accounting information (Birnberg and Shields [1984]). The inability

3 THE ROLE OF VISUAL ATTENTION IN MANAGERIAL JUDGMENT 115 to measure attention could also be the result of cost or time limitations, or difficulty collecting data on attention. 1 Previous BSC studies have focused on the underlying stimulus drawing attention. They assumed that managers attention is linked exclusively to the format of the stimulus. This line of research is primarily concerned about how BSC should be designed or formatted to draw attention (e.g., by using a strategy map (Banker, Chang, and Pizzini [2004, 2011], Cheng and Humphreys [2012]), causal chain (Tayler [2010]), and different formats (Cardinaels and van Veen-Dirks [2010]). The main objective of this line of research is to improve the future design of the BSC. Although useful, these studies ignored the role of informativeness of the stimuli in attracting attention. Recent developments in eye movement based research (e.g., Summerfield et al. [2011], Foulsham and Kingstone [2012]) suggest that visual attention is predominantly driven by the informativeness of the stimuli (goal awareness) rather than its salience (formats). We argue that managers are more likely to focus on strategically linked measures if they are previously aware of their existence, and the company strategy, regardless of the format of the information. Drawing upon psychology and marketing literature, and utilizing the Locarna eye tracker, we developed and tested hypotheses about the role of visual attention in judgment and decision-making. Using the eye-tracking device offers a unique advantage in investigating the use of information in management decisions, compared with an approach relying solely on decision outcomes. An eye-tracking device allows us to investigate how decision quality is affected by the amount of time managers spend looking at linked measures and whether understanding of a firm s strategy and the presentation format affect managers focus. Previous studies relying solely on decision outcomes could not open the black box of how visual attention affects decision quality and what drives visual attention. We argue that visual attention is closely related to improved decision quality. We hypothesize that the relative amount of visual attention that decision makers pay to the cues available to them can be used to predict the quality of their decisions. We also expect that visual attention will be predominantly driven by goal awareness rather than the format of the stimuli (Foulsham and Kingston [2012]). Specifically, we hypothesize that increased attention to strategically linked measures will increase the 1 The cost of an eye-tracking device might be prohibitive to most accounting researchers. The market price for a commercial eye-tracking device is around US$35,000. It is only recently that new software has allowed researchers to analyze eye-tracking data effectively. In the past, researchers have had to analyze eye-tracking data frame by frame, which is very timeconsuming. The cost of eye-tracking devices has decreased significantly in recent years because more companies, including Apple, Microsoft, and Google, are interested in integrating eyetracking features in consumer electronic devices (see, e.g., Luk and Ovide [2013]). Other companies, such as Gazepoint, Eye Tribe, and Tobii, have recently produced more affordable desktop eye trackers.

4 116 Y. CHEN, J. JERMIAS, AND T. PANGGABEAN likelihood of favorable evaluation of those whose achievements are consistent with the strategic objectives of their business unit. We expect the BSC to increase managers awareness of the existence of strategically linked performance measures so that, when they evaluate strategic business unit (SBU) managers, they will pay more attention to these measures. To be consistent with prior studies, we also predict that managers focus on strategically linked measures will be affected by their presentation. To test these hypotheses, we adapted the experimental materials used by Banker, Chang, and Pizzini [2004] to conduct an experiment using the Locarna eye tracker. We asked participants to analyze the performance of two store managers based on 16 performance measures in the BSC. The Locarna eye tracker calculates the participants number of visual fixations and the time spent on each. Fixation is a standard term used in eye-tracking studies, which we define as maintaining an eye gaze on a particular spatial location for an extended period of time (Duchowski [2002]). During the experiment, eye movements were recorded using a lightweight, headmounted eye-gaze tracker manufactured by Locarna Systems. The recorded video streams were postprocessed using the Locarna s Pictus software to produce the eye-gaze coordinates and measure the fixation locations on the items in the BSC. In line with our prediction, we found that managers who paid more attention to strategically linked performance measures were able to evaluate the store managers effectively according to how well their achievements fulfilled the stores strategic objectives. We then investigated what drives attention toward the strategically linked measures, and particularly whether it was because the managers understood the goal (to evaluate the performance of the two SBUs managers) or because of how the information was presented. The eye-tracking device allowed us to obtain evidence related to both the managers awareness of the existence of the strategically linked performance measures (top-down, goal-driven) and how the stimulus is presented (bottom-up, stimulus-driven). We found that managers awareness of the goals was more important than the format of the information in focusing their attention on the strategically linked performance measures. Our study makes four contributions to the accounting literature. First, it provides useful insights into the role of visual attention in judgment and decision-making processes. We found that managers who paid more attention to strategically linked performance measures were more likely to promote staff whose performance was consistent with the strategic objectives of the business unit. Our findings indicate that awareness of the existence of the strategically linked performance measures appears to be more important than their presentation format in improving decisions. Our study also suggests that it is crucial for companies using multiple performance measures and evaluation (such as the BSC) to make managers aware of the relative importance of different measures and encourage focus on the more important cues.

5 THE ROLE OF VISUAL ATTENTION IN MANAGERIAL JUDGMENT 117 Second, our study contributes to theory development about the role of attention in accounting decision research. Prior studies in accounting implicitly assumed the existence of attention in decision-making, but did not observe the attention process itself. According to Birnberg and Shields [1984], researchers assume ex ante a significant relationship between the cue and a decision variable. They also assume that, within an individual s decision-making process, all available cues are considered equally. Attention is an important theoretical construct that should not be ignored, simply because most accounting research is about how information affects decision. Information does not directly affect decisions; it first has to be obtained and processed by the decision maker. Attention serves as a filter, affecting both the type and quantity of information that is used. Understanding the types and quantity of information considered and used by decision makers is crucial to developing a better theory of the role of attention in accounting decision-making. Utilizing a theory of attention, we addressed what affects attention allocation among all pieces of information presented to a decision maker. This will help accounting researchers understand what types of information are more likely to be considered and used in decision-making. Our study investigated whether focusing on more relevant information can lead to better decisions, and under what conditions decision makers focus more on relevant information. Third, our research extends the experiment conducted by Banker, Chang, and Pizzini [2004]. Banker, Chang, and Pizzini [2004] assumed that attention affects performance, but our study was able to directly observe levels of attention by tracking participants visual activities when they performed experimental tasks. We were also able to measure how presentation format affects visual activity. Banker, Chang, and Pizzini [2004, 2011] only implicitly addressed the issue of attention and the weights assigned by managers to different types of performance measure, and did not directly observe how much attention managers paid to strategically linked measures during evaluation. They implicitly assumed that managers put more weight on strategically linked measures, based on the consistency between their evaluations and the linked measures in the experimental materials. With the use of the eye tracker, we were able to obtain firm evidence of both managers awareness of the strategically linked performance measures (top-down, goal-driven) and their response to the way the information was presented (bottom-up, stimulus-driven). This adds to the bottom-up approach of Banker, Chang, and Pizzini [2004]. We agree with Banker, Chang, and Pizzini [2004] that placing more weight on strategically linked measures leads to better decisions, but we argue that decisions are not merely affected by how the stimulus is formatted, but also by awareness of which measures are more important. Our study is the first of which we are aware to directly observe the link between attention and decision in this setting. In other words, instead of relying on the decision to infer managers attention patterns (Banker, Chang, and Pizzini [2004]), we tracked

6 118 Y. CHEN, J. JERMIAS, AND T. PANGGABEAN participants eye movements and related those movements to decisionmaking behaviors. 2 Finally, our study will improve future experimental design in research on attention and decision-making. The eye-tracking technology can capture what is actually happening in the experiment (for instance, in our study, it captures how the subjects allocate their attention). It has the benefit of simplifying the experimental design and improving the logical flow of observation. For example, our results suggest that participants drew more on items on which they fixate for longer. Future research could increase the amount of visual attention that decision makers pay to unique measures in the BSC to mitigate the common measure bias found in previous research (Lipe and Salterio [2000]). The remainder of the paper is organized into five sections. Section 2 presents background theory as the basis for the development of our hypotheses. Section 3 describes the research design and the experimental procedures employed in this study. Section 4 presents the results of the statistical analyses. Section 5 discusses the main findings, limitations, and directions for future research. 2. Background Theory and Hypotheses 2.1. THEORIES OF ATTENTION Attention has been one of the most intensely studied topics in the psychology and marketing literature. Researchers have investigated the topic from a variety of perspectives, such as the determinants of attention (e.g., Yantis and Jonides [1990], Folk, Remington, and Johnston [1992], Öhman, Flykt, and Esteves [2001]), the processes involved (Treisman and Gelade [1980], Wolfe and Horowitz [2004]), and the effects of attention (Spratling and Johnson [2004], Navalpakkam and Itti [2005]). Previous psychology and marketing research indicates that attention is an important factor in understanding individuals decision-making processes. Surprisingly, accounting researchers have not intensively studied attention in general and the role of attention in managers decision-making processes in particular. One possible reason for this is the absence of a device that can be used to track the cues related to how managers process accounting information (Birnberg and Shields [1984]). Kahneman [1973] defined attention as a process of allocating limited processing capacity. Similarly, Carrasco [2011] proposed that attention is 2 Banker, Chang, and Pizzini [2004] used the rating of each manager on a 13-point scale as the measure of likelihood to promote one of these managers, and also asked participants to choose one person for promotion to vice president within the company. We only used the decision to promote one of the two managers. We believe that this is the final outcome of the decision-making process and is more consistent with the information provided in the case indicating that participants ultimate task was to choose which of the two managers to recommend for promotion.

7 THE ROLE OF VISUAL ATTENTION IN MANAGERIAL JUDGMENT 119 a selective process because individuals have a limited capacity to process vast amounts of visual stimuli and must choose which to highlight. Theories of attention can be traced back to the seminal work of James [1890], who proposed that attention involves the mind clearly and vividly focusing on several simultaneously occurring objects and/or trains of thought. According to James [1890], focalization, concentration, and consciousness are the key components of attention. The mind must withdraw attention from some units of information in order to process others more effectively. James s [1890] theory of attention prescribes the determinants of attention to visual stimuli and the influence of attention on communication. Developments of James s [1890] theory of attention suggest that there is a general limit to individuals cognitive capacities. Individuals do, however, have considerable freedom to focus these capacities on different stimuli (e.g., Moray [1967], Kahneman [1973]). Berlyne [1960] suggested that the intensity of attention is largely controlled by the properties of the stimuli to which people are exposed. Researchers have argued that informativeness and salience are the key components of the stimuli that drive attention (e.g., Hyönä, Niemi, and Underwood [1989], Van der Lans, Pieters, and Wedel [2008], Wedel and Pieters [2008]). The informativeness of a stimulus refers to its ability to increase individuals knowledge or reduce their ignorance of the goal to be achieved. Yarbus [1967] argued that the informativeness of stimuli does not reside in the stimuli themselves, but in the goals of the person responding to them. This implies that no stimuli are intrinsically more informative, but rather that stimuli receive more attention when they are perceived to contain information that will help the viewer to achieve a particular goal. The salience of a stimulus is the extent to which it stands out among other stimuli (Lans et al. [2008]). It also describes the attention given to contrasting perceptual features, such as color, texture, size, text, pictures, graphs, places, or people. The strength of the various perceptual features that appear in a stimulus determines the amount of attention the stimulus will receive. Wedel and Pieters [2008] explained that attention is affected by the combination of informativeness and salience. A stimulus will receive the most attention when it is both informative and salient. This attention will, in turn, determine individual preferences, memory (what information is remembered), and choice (what choices an individual makes based on the acquired information). Yarbus [1967] contended that visual attention is predominantly driven more by the informativeness of the stimuli (top-down) than its salience (bottom-up). Recent eye movement based research tends to support this view. For example, Foulsham and Kingstone [2012] investigated how the informativeness of stimuli, as related to the goals (finding a mailbox), and target salience (brightly colored versus homogenous) affected search time. They found that the informativeness significantly affected search time while the salience had little effect. Summerfield et al. [2011] found that, in the

8 120 Y. CHEN, J. JERMIAS, AND T. PANGGABEAN presence of specific goals, the informativeness of the stimuli dominates visual attention. Within an accounting context, theories of attention can be used to explain how the salience and informativeness of various types of accounting information can attract attention and affect decisions. Birnberg and Shields [1984] called for research on attention-related activities, such as tracking, searching, and scanning, and the role of attention and memory in accounting decision-making processes. They claimed that studying these activities is crucial both to the development of a complete understanding of how accounting information is processed by humans and to the interpretation of existing research (p. 365). In this paper, we adopt this theory of attention to explain the role of visual attention in managerial judgments using the BSC performance evaluation. We argue that this theory can explain the determinants of attention to the BSC. Attention to the BSC, as a perceptual object, is driven by awareness of the measures that are important in performing the assigned tasks. The objects in the BSC, including financial and nonfinancial performance measures and strategically linked and nonlinked measures, provide informative stimuli about the different dimensions of the SBU. Attention to the BSC also involves responses to salient stimuli, such as numbers, text, or graphs. 3 The information in the BSC could affect an individual s attention in two ways VISUAL ATTENTION AND THE DECISION-MAKING PROCESS Birnberg and Shields [1984] noted the important role that attention plays in accounting decision-making. They claimed that, within an individual s cognitive processes, attention is affected by cues in accounting information and that attention in turn affects decision-making. They emphasized the importance of identifying which cues are relevant to the decisionmaking process as well as understanding the relationship between cues. The latter will facilitate an understanding of the important role that critical cues play in accountants decision-making processes. Prior marketing studies have concluded that visual attention affects buying decisions. For example, Clement [2007] studied visual influences on in-store buying decisions and demonstrated the effect of package design on consumer attention. This in turn affected in-store buying behavior. Chandon [2002] examined the link between visual attention and brand memory at the point of purchase, and concluded that visual attention is affected by brand recognition, which in turn affects brand choice at the point of purchase. Rosbergen, Pieters, and Wedel [1997] studied the effects of the physical properties of advertising on the visual attention of consumers, which in 3 Many studies in marketing (e.g., Yantis and Jonides [1990], Folk, Remington, and Johnston [1992]) have investigated which characteristics of products, such as shape or color, attract consumer attention. Our study focuses on how presentation format affects attention and how attention affects decisions.

9 THE ROLE OF VISUAL ATTENTION IN MANAGERIAL JUDGMENT 121 turn affects buying behavior. Lohse [1997] investigated how the characteristics of advertisements (colors, size, and graphs) in the Yellow Pages directory affected consumers attention and their choices. Marketing research thus provides significant evidence that buying decisions are positively correlated with consumers visual attention. Findlay, Walker, and Kentridge [1995] proposed that attention is rapidly directed to potentially informative areas of visual displays during eye saccades, and that information is extracted from small regions in the display during eye fixations. Once a person s eye is fixated on a certain point, sufficient time is required for the brain to use this information to increase knowledge about the relationship between the stimuli and the goals (Monk [1984]). Fiske [1980] argued that the duration of fixation time may illustrate how subjects evaluate or comprehend the stimuli. Pieters and Warlop [1999] found that people spent more time looking at chosen brands than nonchosen brands. Fiske [1980] found a positive correlation between the duration of fixation on the stimuli and the relative importance (weight) of each stimulus in decision-making. In this study, we investigate how managers use strategically linked and nonlinked measures and how these measures affect the quality of their decisions. The quality of decisions is defined as managers ability to make decisions that are consistent with the SBU s strategic goals. For example, a manager makes a good quality decision in promoting a member of staff whose achievements are consistent with the SBU s objectives. In alignment with theories of attention, we predict that the amount of time managers spend on strategically linked performance measures will increase the likelihood of promoting the person whose performance is consistent with the business unit s strategic objectives. The following hypothesis will be tested: H1: Managers who devote a large proportion of their time to strategically linked measures will be more likely to promote those whose achievements are consistent with the SBU s strategic objectives VISUAL ATTENTION ON STRATEGICALLY LINKED MEASURES IN THE BSC The informativeness of the BSC is related to how an individual s goals match the information it conveys (Yarbus [1967]). For example, an individual whose goal is to evaluate the SBU s performance based on its strategic achievement will pay more attention to the strategically linked measures. Managers will be able to develop preferences and choices that are consistent with their goals based on the informativeness and salience of the strategically linked performance measures in the BSC. Companies that use the BSC often evaluate their managers according to their achievements on the BSC s key performance indicators (KPIs). Since KPIs are derived from companies vision, mission, and strategies, managers whose performance is linked to KPIs will focus their attention on the strategically linked performance measures. Their managers will also focus more on strategically linked performance measures during evaluations. Those

10 122 Y. CHEN, J. JERMIAS, AND T. PANGGABEAN who perform well on the strategically linked measures should receive a more favorable evaluation than those who perform poorly. The amount of attention on strategically linked performance measures during evaluation should be an important determinant of the quality of decisions. It is therefore crucial to investigate how attention affects the quality of decisions, particularly when managers must evaluate using multiple performance measures, such as in the BSC. Lipe and Salterio [2000] investigated the influence of common measure bias on performance evaluation. They argued that individuals have limited capacity to process information, so ensuring that all measures will be evaluated equally is quite challenging. They predicted that individuals would focus more on common measures (measures common to all business units) rather than on unique measures (measures specific to that individual unit). This simplification strategy is known as common measure bias. In their study, Lipe and Salterio [2000] assigned participants the role of senior managers and asked them to evaluate the performance of managers in two divisions using a BSC for each division. These BSCs included some measures common to both divisions and some specific to one. They found that, when a division performed well on common measures, participants assigned a higher rating to that division than the other. However, when a division performed well on unique measures, there was no difference in the evaluations. Following this study, researchers of BSCs have focused their investigations on how to mitigate common measure bias. Libby, Salterio, and Webb [2004] conducted an experiment similar to that of Lipe and Salterio [2000]. They asked participants to justify their judgment and decision in evaluating the managers of two divisions. They also provided an assurance report stating that all BSC measures were relevant and reliable and should be included in evaluating the two managers. This study showed that providing an assurance report helped participants to include both common and unique measures in their evaluations. This study is based on psychology theory stating that perceived common measure bias could be because of motivational issues in using unique and common measures equally. Other studies have focused on the environmental factors of the BSC, such as its presentation format (Dilla and Steinbart [2005]), the use of strategy maps (Banker, Chang, and Pizzini [2004], Cheng and Humphreys [2012]), varying the BSC format (Lipe and Salterio [2002], Cardinaels and van Veen-Dirks [2010]), and varying the weight of measures (Ittner, Larcker, and Meyer [2003]). These studies showed that environmental factors were significant in mitigating common measure bias. However, they did not incorporate human information processing analysis that could explain the cognitive processes of individuals while using the BSC in performance evaluation. Hunton and McEwen [1997] and McEwen and Hunton [1999] are the only accounting researchers who have used an eye movement retinal imaging system to investigate how the process of tracking and searching for

11 THE ROLE OF VISUAL ATTENTION IN MANAGERIAL JUDGMENT 123 financial information affects financial analysts accuracy. They investigated financial analysts directive or sequential 4 cognitive search strategies and how these strategies related to forecasting accuracy. They concluded that analysts predictions were relatively more accurate when they employed a directive search strategy. Hunton and McEwen [1997] and McEwen and Hunton [1999] also found that analysts experience has less impact on forecasting accuracy than the type of cognitive information search strategy (directive versus sequential) employed. McEwen and Hunton [1999] expanded upon their 1997 study by investigating what types of information in financial statements are used the most by analysts whose predictions are more accurate, and by those whose predictions are less accurate. The researchers concluded that there was a significant difference between the accounting information that the two types of analysts (accurate versus inaccurate) used, and also a significant relationship between the use of accounting information and the associated quality of decision-making. They found analysts who are relatively more accurate take key ratios, earnings summaries, and previous income information into account when they make decisions. Less accurate analysts used more information from footnotes and other items in financial statements. The results from both studies suggest that there is a connection between visual attention and the quality of decision-making. To investigate how visual attention affects the quality of decision-making, researchers require a device that can track the amount of time managers devote to cues perceived as important. Researchers have not previously been able to conduct such investigations either because no such device was available or because its use was prohibitively expensive. For example, Birnberg and Shields [1984] emphasized the need for equipment to measure eye movements to explain attention processes. They believed that an eyetracking device could allow researchers to understand human information acquisition processes and to measure the magnitude of visual attention. The purpose of our study was to investigate how a characteristic stimulus in the BSC, such as strategically linked or nonlinked measures, affects managers attention, and how this in turn affects the quality of their decisions. We used an eye-tracking device to investigate participants scanning processes during a performance evaluation using the BSC. The use of eyetracking allows us to observe the use of the BSC as a performance evaluation system, to monitor the time spent on specific measures, and to examine the relationship between the attention given to specific BSC measures and the quality of decision-making. As the device enables us to investigate managers scanning processes when they evaluate information, we can explore what specific type of information dominates their visual attention and how that affects their decisions. 4 Sequential searchers analyze information by looking at the next item on the screen. Directive searchers analyze information by looking for specific information regardless of the sequence of information that appears on the screen.

12 124 Y. CHEN, J. JERMIAS, AND T. PANGGABEAN Birnberg and Shields [1984] noted the relevance of scanning the act of searching for a set of particular characteristics of stimuli from accounting information to the attention process. We used a BSC containing specific strategically linked and nonlinked performance measures to stimulate the scanning process. We included the nonlinked performance measures since a firm might need to include commonly used and readily available performance measures in performance evaluations even if they were not directly linked to the organization s strategy (Johnson and Kaplan [1987], Ittner and Larker [2003], Bryant, Jones, and Widener [2004], Malina and Selto [2004], Burney and Widener [2007]). In line with theories of attention, we argue that participants attention will be affected by the informativeness of the BSC. We posit that, although all measures in the BSC are important in evaluating the SBU s performance, the achievement of strategically linked measures is relatively more important. This is because these measures are an articulation of the SBU s strategic objectives. Strategically linked measures address managers concern about the lack of value drivers, as well as nonfinancial and forward-looking performance measures, in traditional performance measurement systems (Ittner and Larcker [1998]). The strategically linked measures will attract more attention because they will increase managers knowledge of the SBU s business strategy. This suggests the following hypothesis: H2: When evaluating SBU performance, managers will spend a greater proportion of their time on strategically linked performance measures than nonlinked ones VISUAL ATTENTION AND PRESENTATION FORMAT Tayler [2010] investigated how to mitigate motivated reasoning when the BSC is framed as a causal chain, rather than simply as four independent quadrants. He randomly assigned 135 MBA students into six betweensubjects treatments with two levels of scorecard framing (four group format and causal chain format) crossed with three levels of scorecard implementation involvements (no involvement, strategic selection involvement, and both strategic and measures selection involvement). He found that goaldirected motivated reasoning is mitigated only when the BSC is framed as a causal chain and is combined with managers involvement in the selection performance measures. He showed that simply framing the BSC as a causal chain could not alleviate the effects of motivated reasoning. Cheng and Humphreys [2012] examined the impact of causal linkages between the strategic objectives in a strategy map and organization of performance measures on the ability to recognize relevant information. They conducted two experiments involving 82 graduate students. The students were randomly assigned to a 2 2 between-subjects design that included two levels of strategic objectives (random strategy and strategy maps) and two levels of organization of performance measures (random and

13 THE ROLE OF VISUAL ATTENTION IN MANAGERIAL JUDGMENT 125 scorecard formats). The dependent variables were participants judgments about information relevance and strategy appropriateness. The researchers found that the two elements of the BSC the causal linkages of the strategic objectives in the strategy maps and the categorization of performance measures affected managers information relevance and strategy appropriateness judgments. A structured strategy map improved judgments related to both information relevance and strategy appropriateness, whereas a scorecard format only improved strategy appropriateness judgments. Unlike previous studies focusing on debiasing mechanisms, 5 our study investigates how attention affects judgment and how goal awareness and information formats affect attention. We adapted the research instrument developed by Banker, Chang, and Pizzini [2004, 2011]. Banker, Chang, and Pizzini [2004] argued that strategically linked performance measures help managers to rely less on common measures, and that these measures have a greater impact on evaluation. The researchers found that presenting a firm s strategy in a graphical format to explain the strategically linked performance measures reduced common measure bias in a BSC evaluation. Their argument that a detailed presentation of a firm s strategy improves decision-making performance is based on cognitive fit theory, which suggests that tabular and graphical information improve judgment and decision-making (Vessey [1991]). They do not, however, explain how a graphical presentation affects managers attention, or how that in turn affects their decisions. This limitation is reasonable because the researchers did not employ an instrument to monitor managers attention. We extend the study by Banker, Chang, and Pizzini [2004] by using an eye-tracking instrument to examine the effect of visual attention on managerial judgment. We concur with their argument [2004, 2011] that managers who have detailed information about firm strategies will pay more attention to strategically linked performance measures. Since these measures are an essential component of the BSC, we argue that including a strategic map in our research design will cause managers to focus more attention on strategically linked performance measures. In other words, we predict that presenting information in a graphical format will cause managers to place greater focus on strategically linked performance measures. This is consistent with theories of attention in suggesting that an individual s attention is affected by the salience of different formats (i.e., basic informational, narrative, and graphic representations of the SBU strategy). Figure 1 shows the types of presentation formats in the benchmark, narrative, and strategy map conditions. We therefore developed the following hypothesis: H3: When managers evaluate SBU performance, the difference in the amount of time they spend on strategically linked and nonlinked 5 In BSC literature, the term debiasing is usually used to indicate mechanisms to reduce common measures bias.

14 126 Y. CHEN, J. JERMIAS, AND T. PANGGABEAN FIG. 1. Presentation format of three treatments and the balanced scorecard.

15 THE ROLE OF VISUAL ATTENTION IN MANAGERIAL JUDGMENT 127 performance measures will vary depending upon the format in which information about firm strategies is presented. The difference will be largest for managers with a strategy map, followed by those with a narrative strategy, and, last, by those with basic information about the SBU s strategy. 3. Research Methods 3.1. RESEARCH DESIGN The participants primary task was to evaluate the performance of two SBU managers by comparing their actual achievements with planned performance across 16 performance measures. Participants were then asked to indicate which of the two managers they would recommend for promotion. Participants were randomly assigned to one of three groups, each of which was provided a different format of strategic information (benchmark, narrative, and strategy map). In version 1, participants received a scorecard in which the scores for the Women s Store for the strategically linked measures (across financial, operational, growth, and learning areas) exceeded those of the Family Store. In version 2, participants received a scorecard in which the Family Store had higher scores for each of these four areas. H1 examines the relationship between the amount of time spent on the strategically linked (vs. nonlinked) items and the quality of the associated decision. The amount of time spent on the strategically linked and nonlinked items was obtained from the eye-tracking device. The quality of the decision was defined as whether the promotion decision was consistent with the achievement of the SBU s strategic objectives. We measured the time spent on strategically linked and nonlinked measures in four steps. First, we identified each item in the BSC as strategically linked or nonlinked. Second, the eye-tracking software measured the number of times participants fixated on a particular area of interest (AOI), with fixation defined as spending at least milliseconds ( seconds) 6 focused on an area with a maximum radius of 50 pixels. We designated each performance measure in the BSC as a distinct AOI such that each fixation could only fall within a specific AOI. Figure 2 shows the AOI for the 16 performance measures. The Locarna eye-tracking system has an easy-to-use interface that allows researchers to tag each individual eye fixation. The eye-tracking system recorded eye movements by creating a data file specifying the x y coordinates of each gaze point with a time stamp. The Locarna tagging visualization software processes the raw eye-tracking data and presents each fixation as a cross on a snapshot of a video scene, showing where the participants fix their eyes. We then manually assigned a tag to each fixation 6 Previous studies indicated this as the median of the normal range of fixation length (e.g., Latimer [1988], Rayner [1998, 2009]). This fixation length is also the default setting of the Locarna eye tracker.

16 128 Y. CHEN, J. JERMIAS, AND T. PANGGABEAN FIG. 2. Areas of interest. (see figure A1a and A1b in the Online Appendix). For this study, the tags represent the areas of interest on the experimental materials, such as the customer satisfaction rating on the BSC. The fixation time associated with each tag was saved to an Excel file, and the tagged data used for further statistical analysis. To ensure the accuracy of the tagging, each video was tagged by one researcher and verified by another. We summed the time the participants spent on all the strategically linked and nonlinked items, then calculated the proportion of time spent on the strategically linked and nonlinked items. The software used by the Locarna eye-tracking device is able to create a heat map based on the time spent on each item. The heat map is a plot of the intensity of participants visual attention. To create the heat map shown in figure 3, the fixation points were mapped into a picture of a BSC based on the data generated by the eye-tracking equipment. The rainbow palette was used in the plotting. The white color areas have more fixation points and a longer fixation time, which indicates that the participants looked at the areas more frequently and for a longer period of time than they did at the orange and blue areas. In this study, as shown in figure 3, participants spent more time looking at the linked items (white areas) than at the nonlinked items (orange and blue areas).

17 THE ROLE OF VISUAL ATTENTION IN MANAGERIAL JUDGMENT 129 FIG. 3. Heat map. The heat map is a plot of the intensity of the visual attention. The fixation points were mapped into a picture of the BSC based on the data generated by the eye-tracking equipment. The areas on the BSC with more fixation points and longer fixation times are shown in a hotter color. The rainbow color palette was used in the plotting. The hotter color areas indicate that the participants looked at the areas more frequently and for a longer period of time than they did at the other color areas. We used a chi-square procedure to test H1, which enabled us to examine whether the distribution of the categorical variable (i.e., the decision to promote a particular store manager) differed between participants who spent more or less time on strategically linked measures. H2 and H3 are about how managers arrive at promotion decisions. To test these hypotheses, we examined the amount of time participants spent on linked and nonlinked measures, and the effects of SBU strategic information format on the proportion of time spent on strategically linked versus nonlinked measures. The design of the experiment was a 2 3 repeated measure mixed analysis of covariance (ANCOVA) design, with one between-subject and one within-subject factor (independent variable). The between-subjects factor was the format of SBU strategic information (benchmark, narrative, and strategy map), and within-subject factor was the favored measures (linked or nonlinked). The dependent variable was the proportion of time spent on the strategically linked and nonlinked

18 130 Y. CHEN, J. JERMIAS, AND T. PANGGABEAN performance measures. We used gender and length of work experience as covariates since previous studies have indicated that these might influence the time spent evaluating managers PARTICIPANTS Employees from several accounting and multinational corporations were recruited to participate in this study. We contacted the senior management of the corporations to ask if their employees were willing to participate. Employees who were interested in participating were asked to contact us via or telephone to arrange a suitable time to do the experiment. The experiment was administered to participants one at a time based on the availability of the eye-tracking device. Participation was voluntary, and participants were told that they could discontinue the experiment at any time without penalty. They were informed that they were going to participate in a study using an eye-tracking device to explore managers judgments in evaluating SBU performance. Participants took an average of 20 minutes to complete the experiment and were compensated with a $10 gift certificate. A total of 60 employees participated. Their average age was 25, and the average length of their work experience was five years. Participants indicated that they had visited a retail clothing store an average of three times in the previous 12 months. Around two-thirds, or 63%, of the participants were female and 37% were male APPARATUS A Locarna eye-tracking system was the primary input device (see figures A2 and A3 in the Online Appendix). The eye tracker was connected to a conventional laptop PC running Microsoft Windows XP. The system s display was a 11 LCD. The eye tracker sampled at 30 Hz with an accuracy of visual arc, or about pixels with the configuration used. Raw eye-tracking data were collected and analyzed using the Locarna s Pictus software. Participants were asked to wear the eye-tracking device. We then placed a15cm 12 cm paper with nine black dots on the table in front of each person. The distance between the paper and the participants was normal reading distance, about cm. The paper was placed in such a way that the nine dots corresponded exactly to the nine boxes on the computer screen. Once the dots were fixed with the computer screen, we asked participants to fix their eyes on each dot in turn to capture the eye position for each. A snapshot of the calibration process is shown in figure A4 in the Online Appendix. 7 7 An Online Appendix to this paper can be downloaded at edu/arc/journal/onlineappendices.aspx.

19 THE ROLE OF VISUAL ATTENTION IN MANAGERIAL JUDGMENT EXPERIMENTAL MATERIALS AND PROCEDURES The experimental materials used in this study were adapted from Banker, Chang, and Pizzini [2004]. We asked participants to assume the role of a senior executive at a major clothing retail chain with over 3,000 stores, generating US$6 billion in annual sales. Participants learned that each store was organized as a separate SBU with a distinct image and target market. The materials indicated that the company had recently implemented a new performance measurement system based upon the BSC to replace its previous financially oriented system. Participants were randomly assigned to one of three groups. In the benchmark condition, participants were provided with a description of the company s overall goals and an overview of the strategy of the two business units (Family Store and Women s Store). The strategy descriptions in the benchmark condition were similar to those used by Lipe and Salterio [2000]. Participants were also told that the Women s Store was in the mature stage of its business lifecycle, while the Family Store was in the growth stage. In the narrative condition, participants were provided with a brief elaboration of the strategy of the two business units, as well as the overview provided to participants in the benchmark condition. In the strategy map condition, participants received a simple strategy map as well as the information provided to narrative condition participants. The participants were given the two business units BSC performance measures and targets. They were then given the two SBUs performance reports, which showed their targets, realized performance for that period for all 16 performance measures, and the percentage of actual performance against the targets. 8 Participants wore the eye-tracking device throughout their evaluation of the case. The device is very light (30 g), portable, and unobtrusive. Following Banker, Chang, and Pizzini [2004], we specified four measures for each of the perspectives in the BSC. For each, we used two performance measures that were directly linked to the SBU s strategy and two that were not directly linked, giving eight strategically linked and eight nonlinked measures. Consistent with Lipe and Salterio [2000], both SBUs performed better than target on all 16 items. 9 We used two different scorecard versions from Banker, Chang, and Pizzini [2004] to represent the relative performance of the two SBUs on strategically linked and nonlinked measures. 10 Table 1 shows the two versions of the BSC and the 16 items used. 8 We randomly varied the order in which the participants saw the stores to mitigate potential recency effects. 9 We assumed that both business units were in the same financial position (short-term condition), and neither was in trouble on these (short-term) financial measures. This assumption is consistent with the fact that both business units performed better than their targets for all 16 measures. Although all the measures are important (Kaplan and Norton [1996]), the performance on strategically linked measures is relatively more important. 10 Versions 1 and 2 were the same as versions 6 and 11 in Banker, Chang, and Pizzini [2004].

20 132 Y. CHEN, J. JERMIAS, AND T. PANGGABEAN TABLE 1 Versions of the BSC and Types of Performance Measures Panel A: Types of Measures BSC Version The Women s Store The Family Store 1. L N 2. N L Panel B: The Women s Store Type The Family Store Financial: Financial: 1. Sales margins L 1. Sales margins 2. Sales growth per store L 2. Percentage of sales from new stores 3. Inventory turnover N 3. Operating cash flow as a percentage of sales 4. Debt-to-assets ratio N 4. Debt-to-assets ratio Customer: Customer: 1. Price relative to competitors price N 1. Price relative to competitors price 2. Customer satisfaction rating L 2. Customer satisfaction rating 3. Sales per square foot of retail space L 3. Percentage of sales to new customers 4. Number of credit card customers per store N 4. Number of new items in which first to market Internal Process: Internal Process: 1. Brand recognition rating L 1. Market share per advertising dollar 2. Number of stock-outs N 2. Vendor rating 3. Mystery Shopper audit rating L 3. Mystery Shopper audit rating 4. Time to process customer returns N 4. Time to process customer returns Learning and Growth: Learning and Growth: 1. Employee satisfaction L 1. Employee satisfaction 2. Employee suggestions per year N 2. Sales associates with college degrees 3. Store computerization N 3. Store computerization 4. Hours of training invested in brand managers each year L 4. Retail experience of marketing managers L, linked; N, nonlinked. Versions 1 and 2 are the same as versions 6 and 11 in Banker, Chang, and Pizzini [2004]. We decided not to replicate the entire Banker, Chang, and Pizzini [2004] design because the main purpose of our study was to investigate whether the BSC helps managers to focus more on strategically linked measures and how attention affects decisions. The two versions of the BSC allowed us to examine a business setting in which performance on strategically linked measures favored one store and performance on strategically nonlinked measures favored the other (hence the term favored measures, describing whether performance was better on strategically linked or nonlinked measures in that version of the BSC). This research design enabled us to test the hypothesis that the BSC helps managers to focus on strategically linked measures, because the expectation that managers will pay more

21 THE ROLE OF VISUAL ATTENTION IN MANAGERIAL JUDGMENT 133 attention to strategically linked measures is unequivocal when performance at both stores favors the linked measures. The average percentage above the target, however, was similar for both stores (8.56%). In one version, the Women s Store performed better on the strategically linked measures and the Family Store on the nonlinked measures. In the other version, the Family Store was better on the strategically linked measures, and the Women s Store on the nonlinked measures. After participants had reviewed the performance reports for both stores, they were asked to decide whether to promote the manager of the Women s Store or the Family Store, using the following question: Based upon the information you have, which of the two managers, Chris Peters (The Women s Store) or Taylor Graham (The Family Store) would you recommend for promotion? Participants put a check mark next to their chosen manager s name. They were then asked to complete a postexperiment questionnaire and provide demographic information. 4. Data Analyses and Results 4.1. MANIPULATION CHECK To assess whether narrative or strategy map improved understanding of the SBU s strategy, we asked participants to rate their level of agreement with four statements about their understanding, on a 9-point scale (from 1 = strongly disagree to 9 = strongly agree). These four statements were: 1. The strategy of the Women s Store is to generate greater sales through its existing infrastructure rather than invest in new stores. 2. To grow sales, the Women s Store must successfully introduce new lines of clothing to its existing customers. 3. The strategy of the Family Store is to grow by adding new stores. 4. The Family Store needs an innovative marketing group because its growth plans depend upon the success of its advertising campaign in attracting new customers. Table 2 shows the statements about strategy comprehension and the average responses across the groups. Overall, participants in the narrative and strategy map groups gave higher ratings to the four statements, though only one statement was statistically significant. The average scores for the four strategy comprehension statements were 6.79, 7.19, and 7.05 for the benchmark, narrative, and strategy map treatments. We tested the internal consistency between the four statements of strategy comprehension using Cronbach s alpha, which was 0.56, indicating internal consistency among the four statements. We also assessed participants overall understanding of the task by asking them to rate their agreement with seven statements on task understandability using a 9-point scale (where 1 = strongly disagree to 9 = strongly agree; see panel B, table 2). We asked participants whether it was appropriate

22 134 Y. CHEN, J. JERMIAS, AND T. PANGGABEAN TABLE 2 Mean Responses of Each Question Related to Strategy and Task Understanding Panel A: Strategy comprehension 1. The strategy of The Women s Store is to generate greater sales through its existing infrastructure rather than invest in new stores. 2. To grow sales, The Women s Store must successfully introduce new lines of clothing to its existing customers. Two-Sample Mean Mean Test t-test (p < 0.1) Benchmark Benchmark Narrative Benchmark Narrative Strategy Map Strategy Map Narrative Strategy Map The strategy of The Family Store is to grow by adding new stores The Family Store needs an innovative marketing group because its growth plans depend upon the success of its advertising campaign in attracting new customers. Average strategy comprehension score Panel B: Understanding of task 1. The performance measures were usefully categorized in this case Financial performance measures were emphasized in this case The two business units, The Women s Store and The Family Store, used some different performance measures. 4. It was appropriate for the two business units, The Women s Store and The Family Store, to employ some different performance measures The case was easy to understand The case was difficult to do The case was realistic Average understanding of task score We asked participants to indicate their degree of agreement with each question on a 9-point scale (from 1 = strongly disagree to 9 = strongly agree). The null hypothesis is that means for both treatments are equal. The null hypothesis is rejected at a significance level of p > and denote a significance at 0.10, and 0.05 (two-tailed).

23 Panel A: THE ROLE OF VISUAL ATTENTION IN MANAGERIAL JUDGMENT 135 TABLE 3 Descriptive Statistics of Time Spent on Linked and Nonlinked Performance Measures Descriptive Statistics (in seconds) Benchmark Narrative Strategy (n = 20) (n = 20) Map (n = 20) Total Variables Mean SD Mean SD Mean SD N Mean SD Linked fixation Nonlinked fixation Gender Experience Panel B: Mean Presentation Linked Nonlinked Format Fixation Ratio Fixation Ratio t-value p-value Overall a Benchmark ns Narrative ns Strategy Map ns a The sum of ratio of time spent on linked and nonlinked measures does not add up to 1 since there were six participants who did not spend enough time on these items to be captured by the eye-tracking device (i.e., zero fixation time for both linked and nonlinked measures). When we excluded these six participants, the sum of the ratios was equal to 1 and thet-test results were qualitatively similar to those reported in panel B. The reason why we included these six is because they did look at the experimental materials and complete the tasks, including making the promotion decision. We also found out that the manipulation treatments were successful for these six participants, indicating that they understand the experimental materials. We investigated these six participants further by checking the video recorded by the eye-tracking device. Based on this analysis, we found that each of them actually spent time looking at the area of interest in the BSC. However, they did not look long enough at any area of interest to be considered a fixation based on our criteria of milliseconds. Previous eye-tracking studies have indicated that people can extract meaningful information within the range of milliseconds (Rayner [1998]). It seems that these six participants only needed a relatively short time to extract meaningful information from the experimental materials. and denote a significance at 0.10, and 0.05 (two-tailed). for the two stores to have different performance measures, and whether the tasks were realistic and easy to understand. Overall, participants in the narrative and strategy map groups gave higher ratings to the seven statements, though only two questions were statistically significant. The average scores for the seven task understandability statements were 6.23, 6.48, and 6.73 for the benchmark, narrative, and strategy map treatments. Overall, the results indicate that our narrative and strategy map treatments were successfully manipulated. We tested the internal consistency of the overall understandability of the tasks among the seven statements using Cronbach s alpha, which was 0.55, indicating internal consistency among the seven statements DESCRIPTIVE STATISTICS Table 3 reports the descriptive statistics for the first stage of the experiment. Panel A shows the descriptive statistics of the length of time (in

24 136 Y. CHEN, J. JERMIAS, AND T. PANGGABEAN FIG. 4. Fixation time spent on linked and nonlinked performance measures (error bars indicate standard error). Participants in the benchmark group spent the most time on the strategically linked performance measures, followed by participants in the narrative group, and in the strategy-map group. Participants in the benchmark group spent the least time on the strategically nonlinked performance measures, followed by participants in the narrative group, and those in the strategy-map group. seconds), while panel B shows the ratio of time spent on linked and nonlinked performance measures, partitioned by the format of supplemental information provided to participants. Participants in the benchmark group spent the highest proportion of time on the strategically linked performance measures, followed by the narrative group, and then by the strategy map group (mean benchmark = 0.51, mean narrative = 0.49, and mean strategy map = 0.47). Participants in the benchmark group also spent the lowest proportion of time on the nonlinked performance measures, followed first by the strategy map group and then by the narrative group (mean benchmark = 0.39, mean narrative = 0. 44, and mean strategy map = 0.43). These results are shown graphically in figures 4 and 5. One concern with the use of multiple performance measures, such as the BSC, is that managers might only pay attention to familiar measures and consequently not fully utilize the BSC in decision-making. We therefore examined the pattern of the time participants spent on linked and nonlinked measures across the treatments. As shown by table 4, participants focused on all linked and nonlinked measures and the pattern was consistent across the groups, even though the length of time spent on each measure was different We rank ordered the items based on the amount of time participants spent on each, to investigate whether a particular item drove the decision. We compared the impact of the highest

25 THE ROLE OF VISUAL ATTENTION IN MANAGERIAL JUDGMENT 137 FIG. 5. Results of the effect of supplemental strategic information on the proportion of time spent on strategically linked and nonlinked performance measures (error bars indicate standard error). Participants in the benchmark group spent most time on the strategically linked performance measures, followed by participants in the narrative and strategy-map groups. Participants in the benchmark group spent the least time on the strategically nonlinked performance measures, followed by participants in the narrative group and in the strategy-map group HYPOTHESIS TESTING We used a chi-square procedure to test the effect on decision quality of the length of time spent on strategically linked performance measures. The dependent variable was the quality of the decision. We used a meansplit approach 12 to classify the sample into those who spent more and less time on strategically linked measures. We obtained two groups of 33 who spent more time on strategically linked measures, and 27 who spent less. H1 predicts that managers who spend a high proportion of their time on strategically linked measures will make better decisions than those who spend a low proportion of their time on these measures. Table 5, column 2 indicates that managers who spent more time looking at strategically linked and lowest ranked items on the quality of participants decisions. The results (not reported for brevity) show that the impact of the two was relatively similar. These results indicate that participants decisions were not being influenced by any particular item in the BSC, but were affected by all of them. We also compared the four highest and lowest ranked items. The results (not reported for brevity) were qualitatively similar to those using the highest and lowest ranked items. 12 We also used a median-split approach and the results (reported in table A1 in the Online Appendix) were qualitatively similar to those using a mean-split approach.