Auditors Reactions to Inconsistencies between Financial and Nonfinancial Measures

Transcription

1 Auditors Reactions to Inconsistencies between Financial and Nonfinancial Measures Joseph F. Brazel* Reznick Group Research Fellow Department of Accounting College of Management North Carolina State University Campus Box 8113 Raleigh, NC Telephone: Fax: Keith L. Jones George Mason University Douglas F. Prawitt Glen Ardis Professor of Accountancy Brigham Young University April 2012 We are grateful for the helpful comments provided by Chris Agoglia, Paul Beswick, Brian Croteau, Steve Glover, Erin Hamilton, Rick Hatfield, Rich Houston, Kathryn Kadous, James Kroeker, Kathleen Linn, Linda Parsons, Mark Peecher, Sean Peffer, Dave Piercey, Dick Riley, Chad Simon, Gary Taylor, Rick Warne, Jeff Wilks, David Wood, and Mark Zimbelman. We also thank workshop participants at the University of Alabama, the 2009 Accounting, Behavior and Organizations Research Conference, the 2010 Auditing Mid-Year Conference, the 2010 Public Company Accounting Oversight Board Academic Conference, the 2010 Inaugural Conference on Financial Reporting, Auditing and Governance (winner of the best paper award), the 2010 International Symposium on Audit Research, the 2010 AAA Annual Meeting, the 2010 Annual Meeting of the Institute for Fraud Prevention, the 2012 Mid-Atlantic Region Conference for the Institute of Internal Auditors, the 2012 Meeting of the Association of Certified Fraud Examiners Central Carolina Chapter, the Raleigh-Durham chapter of the Institute of Internal Auditors, the Research Center on the Prevention of Financial Fraud, and the Office of the Chief Accountant of the Securities and Exchange Commission. We appreciate the support of the audit professionals who participated in this study and the research assistance of Kylee Phillips and Marycobb Randall. This research was supported by a grant from the Financial Industry Regulatory Authority (FINRA) Investor Education Foundation. All results, interpretations, and conclusions expressed are those of the authors alone, and do not necessarily represent the views of the FINRA Investor Education Foundation or any of its affiliated companies. * Corresponding Author

2 Auditors Reactions to Inconsistencies between Financial and Nonfinancial Measures ABSTRACT: Nonfinancial measures (NFMs), such as employee headcount and production space, are operational measures that are not included on the face of the financial statements but are often disclosed elsewhere in the annual report or 10-K (e.g., in Management s Discussion and Analysis). Professional standards, auditing texts, and prior research suggest that external auditors can use NFMs to verify their clients reported financial information and, in turn, improve audit quality. In an initial experiment where auditors were asked to develop an expectation for a client s sales balance, they generally failed to identify a seeded inconsistency between the client s sales and related NFMs. In a second experiment, we find that auditors are more likely to react to the inconsistency (i.e., rely more on inconsistent NFMs / develop expectations that reflect the client s current year decline in NFMs) when they are specifically prompted to consider the implications of NFMs and fraud risk is high (vs. low). Our results suggest that (1) a minority of auditors use NFMs as a source of information for testing and do not increase their reliance on NFMs when the NFMs point to a fraud red flag; (2) the presence of high fraud risk alone is insufficient to increase auditor consideration of inconsistent NFMs; (3) auditors are able to react appropriately to an inconsistency if they are effectively prompted; and (4) even a prompt that explicitly highlights an inconsistency is unlikely to improve auditor performance unless the prompt is processed in a high fraud risk setting. Keywords: analytical procedures, audit, fraud, nonfinancial measures Data Availability: Data are available upon request.

3 INTRODUCTION Capital market participants and other stakeholders in the financial reporting process indicate that fraud detection is among their highest priorities (Elliott 2002; PCAOB 2003, 2007). Professional standards (e.g., AICPA 2002), auditing texts (e.g., Messier et al. 2012), and prior research (e.g., Brazel et al. 2009) suggest that an inconsistency between a company s financial performance and related nonfinancial measures (e.g., number of retail outlets, warehouse space, employee headcount) represents a potential red flag with respect to financial statement fraud. 1 For example, lawsuits against the auditors of Bernard L. Madoff Investment Securities LLC and its feeder funds claim that the auditors could have identified the fraud by noting the inconsistency between Madoff s bloated balance sheet and the firm s minimal trading volume (Dugan and Crawford 2009; Lauricella 2009). The Public Company Accounting Oversight Board (PCAOB) has discussed the potential for nonfinancial measures (NFMs) to provide a powerful, independent benchmark for evaluating the validity of financial statement data and has endorsed the use of NFMs to improve fraud detection (PCAOB 2004). A recent archival study tests these conjectures and finds that, for fraud firms, growth in revenue significantly exceeds growth in related NFMs (Brazel et al. 2009). By contrast, Brazel et al. (2009) and other studies find NFMs to be relatively consistent with financial statement data for non-fraud firms (e.g., Ittner and Larcker 1998). Thus, there is reason to believe that auditor consideration of inconsistencies between financial statement data and NFMs could improve the detection of fraudulent financial reporting (hereafter, fraud). Such 1 We use the term inconsistency to describe the inconsistencies between financial data and related NFMs that fraud firms tend to exhibit. For example, for fraud firms, Brazel et al. (2009) find that revenue growth exceeds NFM growth by approximately 25% (inconsistency) whereas revenue growth exceeds NFM growth by approximately 10% for non-fraud firms (consistency). 1

4 improvements are needed as recent evidence suggests that the media, employees, analysts, and short sellers are all more likely than the auditor to detect fraud (Dyck et al. 2011). Despite the important role NFMs can play in fraud detection, archival and survey research come to different conclusions regarding the extent to which auditors focus on NFMs when evaluating financial data. Results from recent archival studies raise the possibility that auditor attention to NFMs is insufficient to detect inconsistencies between financial data and NFMs (Brazel et al. 2009; Dechow et al. 2011). On the other hand, survey studies of auditors point to increased NFM usage from the 1990 s to the 2000 s (Hirst and Koonce 1996; Trompeter and Wright 2010). Our first objective is to leverage the experimental method to explore the extent to which auditors identify and question inconsistencies between financial data and related NFMs. Our second objective is to investigate if and how auditors reactions to such inconsistencies can be improved. Guided by prior empirical findings related to auditor use of NFMs and the Heuristic- Systematic Model (e.g., Chaiken 1980; Dechow et al. 2011), we posit that: (1) auditors generally do not react (e.g., rely more on inconsistent NFMs) to inconsistencies between financial data and related NFMs; (2) a prompt can improve auditor attention to such inconsistencies; and (3) the effectiveness of the prompt is dependent on the fraud risk assessment for the audit engagement. We test these conjectures in two experiments with a total of 110 in-charge senior auditors. Our first experiment places auditors in one of two conditions: a hypothetical client s NFMs are either consistent or inconsistent with their reported revenue account balance. Participants performed a substantive analytical procedure related to revenue. We provided them with a variety of data to use when developing their expectation for revenue (e.g., prior year financial data, budgeted financial amounts, NFMs). Results for Experiment 1 indicate that a 2

5 minority of participants used NFMs and that participants did not rely more heavily on NFMs in the presence of an inconsistency. Importantly, under the inconsistent NFM condition, where the NFM data indicates the possibility of a material overstatement of revenue, we find that auditors expectations did not reflect the inconsistency. Only 1 of the 21 auditors in the inconsistent NFM condition provided an expectation that was materially different from the client s revenue balance (i.e., 20 of 21 auditors concluded that the account balance was reasonable). Our second experiment mirrors the first except that (1) all conditions involve an inconsistency between revenue and NFMs; (2) we manipulate fraud risk between participants as low or high; and (3) after developing an initial expectation for revenue, we prompt participants to explicitly consider the impact of NFMs and then ask participants to provide a final expectation (a within-participant manipulation). We find no direct effect for the fraud risk manipulation on auditors NFM reliance or expectations, indicating that heightened awareness of the possibility of fraud did not serve to focus auditors on the seeded inconsistency. Further, while the prompt makes auditors more likely to rely on NFMs and question the client s revenue balance, the effect of the prompt depends on the level of fraud risk. Auditors in the high (vs. low) fraud risk setting were more likely to develop a final (i.e., post-prompt) expectation that would lead to further questioning, testing, and other activities that are prerequisites for assessing the reasonableness of the inconsistency. Greater attention to the prompt under high fraud risk would perhaps be appropriate and efficient if auditors were consistently effective at assessing fraud risk. Prior experimental studies find that auditors fraud risk assessments tend to reflect the components of the fraud risk triangle / fraud cues (e.g., Zimbelman 1987; Asare and Wright 2004). For example, Wilks and Zimbelman (2004) illustrate what when opportunity and incentive cues suggest high fraud risk 3

6 (e.g., significant related party transactions), auditors fraud risk assessments are elevated. However, recent field research finds that auditors, ex ante, fail to assess fraud risk higher for companies where fraud is later detected (Brazel et al. 2010). Similarly, Dyck et al. (2011) observe that auditors are responsible for only 10 percent of corporate fraud revelations. Employees and the media, who are not charged with detecting material fraud / assessing fraud risk, reveal 17 percent and 13 percent of frauds, respectively. It is possible this low detection rate for auditors is related to inaccurate fraud risk assessments during audit planning. Inaccurate fraud risk assessments could, in turn, lead to under- or over-auditing during the substantive testing phases of audits. In short, it is possible that some fraud risk assessments are not well calibrated. Thus, a tendency to discount fraud red flags revealed via substantive tests when fraud risk is assessed at low may be problematic. This study contributes to the literature and practice in several ways. First, the primary objective of an audit is to determine if a company s financial statements fairly reflect the economic events of the company (e.g., Bell et al. 2005; Arens et al. 2010). If NFMs act as surrogate measures of economic activity or performance (Schultz et al. 2010), our finding that auditors do not react to an inconsistency between financial data and related NFMs suggests a need for improvement in this vital area. Related to this issue, our study provides evidence that a relatively simple and efficient prompt can improve auditor performance in the important area of revenue recognition. However, we find that the prompt is more effective when fraud risk is assessed as high during planning. Second, we contribute to a literature that has mainly focused on fraud-related planning tasks (i.e., fraud hypothesis generation, risk assessment, and audit program modification) versus the evidence evaluation that occurs during substantive testing (e.g., Carpenter 2007; Hogan et al. 4

7 2008; Hammersley et al. 2010; Hammersley et al. 2011; Hammersley 2011). We examine how fraud risk assessments and the incorporation of a prompt can influence how auditors respond to evidence of a red flag v i s - à - v i s a substantive analytical procedure. We provide evidence that, unless fraud risk is assessed as high during planning, auditors may not respond to red flags explicitly identified by a prompt. Third, we document that a well-established theory from psychology (the Heuristic-Systematic Model) can be used to explain auditor behavior in the crucial area of red flag analysis. This paper is organized as follows: Section two develops our hypotheses, and Section three explains our research methods. Section four provides the results, and Section five concludes the paper. DEVELOPMENT OF HYPOTHESES The role of NFMs in financial statement auditing A growing body of literature offers evidence that NFMs can provide a valuable benchmark for evaluating companies financial statements (Ittner and Larcker 1998; Bell et al. 2005; Knechel 2007; Knechel et al. 2010; Schultz et al. 2010). In auditing, SAS No. 56 (AICPA 1988) suggests that auditors should consider NFMs when performing analytical procedures. Auditors use analytical procedures to assess risks and to detect material misstatements (AICPA 1988; IAASB 2010; AICPA 2007). For example, when developing an expectation for revenue, an auditor could consider growth in the number of client retail outlets. To serve as an effective benchmark for evaluating financial statement data, NFMs must correlate with the performance reflected in the financial statements. A stream of studies concludes that such relations exist. For example, Ittner and Larcker (1998) find that one form of NFM customer satisfaction is significantly related to future accounting performance and is 5

8 partially reflected in current accounting book values. Consistent with this research, professional audit guidance suggests that auditors can expect NFMs such as production capacity to be correlated with revenue reported on the income statement (AICPA 1988; AICPA 2002). Brazel et al. (2009) add to this research stream by finding that fraud firms exhibit inconsistencies between their revenue growth and contemporaneous growth in NFMs. For fraud firms, revenue growth substantially exceeds growth in related NFMs. However, for a matchedsample of non-fraud competitors, the relation between revenue and NFM growth is quite consistent (see footnote 1 for additional information). Identifying inconsistencies between financial data and related NFMs should serve as a red flag for auditors, leading them to ask pointed questions of client management, increase their professional skepticism, corroborate and test management s responses with reliable evidence, assign forensic specialists to the engagement, and increase the overall likelihood of fraud detection. When performing analytical procedures to detect material misstatements, auditors must be wary about relying too heavily on financial data (e.g., prior year account balance, industry revenue growth) because management can often manipulate current year financial data to create an expected pattern (PCAOB 2004). Accordingly, some researchers suggest that auditors should rely more on information sources that are less prone to manipulation by management (Wilks and Zimbelman 2004). In this vein, NFMs (e.g., production facilities, retail outlets, and employees) are typically less vulnerable to manipulation and are often easier to verify than financial data (Bell et al. 2005). In addition, many NFMs are produced and reported by independent sources (e.g., trade groups, etc.). Finally, many NFMs should be fairly easy for auditors to access because companies maintain NFM data for disclosure in their 10-K filings (Brazel et al. 2009). 6

9 NFMs and analytical procedures Analytical procedures consist of developing an expectation for a company s reported financial statement data (e.g., account balance or ratio), comparing the expectation to the financial statement data, and investigating any significant differences. While the results of Brazel et al. (2009) suggest that NFMs could be used during planning to more accurately assess fraud and business risks, the aggregation of data and lack of testing at the planning stage make it difficult to actually detect a material misstatement using preliminary analytical procedures. On the other hand, analytical procedures used during substantive testing are becoming a more important component of audit testing (Trompeter and Wright 2010). They can appropriately serve as the primary tests of the income statement when they are performed at a disaggregated level (e.g., division or product line, etc.). During substantive testing, significant differences between auditor expectations and financial data are resolved through additional inquiry and evidence accumulation. These procedures can provide meaningful audit evidence and can identify material misstatements due to either error or fraud (AICPA 1988; Hirst and Koonce 1996). 2 Why NFMs are a unique and powerful evidence source Current audit guidance for publicly traded companies explicitly states that if audit evidence obtained from one source is inconsistent with that obtained from another, or if the auditor has doubts about the reliability of information to be used as audit evidence, the auditor should perform the audit procedures necessary to resolve the matter and should determine the effect, if any, on other aspects of the audit (PCAOB 2010, paragraph 29). SAS No. 56 (AICPA 1988) lists the following information sources for developing an expectation during analytical 2 On the other hand, if the auditor s expectation for revenue and the client account balance are not materially different, the auditor typically does not perform any additional analysis as the immaterial difference is considered evidence that the client s account is reasonably stated (AICPA 1988; Arens et al. 2010). 7

10 procedures: financial data from prior periods, client financial budgets, relations between financial information in the current period (e.g., ratio or regression analyses), industry financial data, and relations between financial information and relevant NFMs. As already noted, an inconsistency between revenue and related NFMs can be considered a fraud red flag. However, with regard to the other (non-nfm) information sources, prior research suggests that consistencies between financial statement data and these sources may well exist in a fraud setting. For example, if a company is smoothing earnings, then current year financial data will be consistent with prior year data (Trompeter and Wright 2010). If a company is managing earnings to meet its budget or to be consistent with industry trends, then current year financial data will be consistent with budget or industry data. Along these lines, researchers observe that both budgetary and industry pressures are important incentives when auditors apply the fraud triangle in practice (e.g., Brazel et al. 2010). Also, results from prior studies suggest that fraud firms have a tendency to manipulate financial data to be consistent with their industry (e.g., Beneish 1997; Summers and Sweeney 1998; Lee et al. 1999). The PCAOB has noted that analytical procedures using only financial data (ratio analyses and regression) are likely to be ineffective for detecting fraud because management can make fictitious entries to financial data in order to create an expected or consistent pattern (PCAOB [2004]). In short, these consistent patterns can be problematic from an audit perspective as they will yield auditor expectations that confirm the company s account balances and require no further attention from the auditor (see footnote 2). Therefore, from a fraud detection perspective, prior research suggests and the PCAOB (2004) contends that NFMs can serve as a unique and powerful evidence source. 8

11 Do auditors react to inconsistencies between financial and nonfinancial measures? In a field investigation, Hirst and Koonce (1996) find that auditors typically follow a simplistic process when developing expectations for substantive analytical procedures. Specifically, auditors tend to rely heavily on prior year account balances when developing expectations for current year balances. They report moderate use of client budgets, minimal use of regression analyses, and no use of NFMs. Complementing Hirst and Koonce (1996), Trompeter and Wright (2010) investigate the information sources that auditors use when developing expectations for analytical procedures. Trompeter and Wright (2010) point to environmental changes over the past decade that should have led auditors to use more sophisticated methods when developing expectations (e.g., technological advancements and greater access to information). With respect to ordering the importance of information sources currently used, their survey results are as follows: (1) prior year balance, (2) industry data, (3) NFMs, (4) client budgets, (5) interim data, and (6) regression. Consistent with Hirst and Koonce (1996), auditors still report that they rely heavily on prior year balances when developing expectations (in which case it is unlikely that they will respond to inconsistencies be financial data and related NFMs). However, it appears that auditor use of NFMs has increased from the 1990 s to the 2000 s, which introduces the possibility that auditors are sensitive to such inconsistencies. Trompeter and Wright (2010) provide several examples of auditors using NFMs to verify financial statement data. In line with this apparent trend, Cohen et al. (2000) illustrate that auditors consider nonfinancial information when developing reasons for why some financial accounts need additional audit effort. Auditor NFM reliance may have also increased since Hirst and Koonce (1996) because auditors can be trained in the ways NFMs can be used to verify financial data. 9

12 For example, Brewster (2011) provides evidence that undergraduate accounting students can be trained to effectively incorporate NFMs into their judgments. Still, recent archival research documents that fraud firms typically exhibit inconsistencies between their financial data and related NFMs. Brazel et al (2009) find that revenue growth for fraud firms differs substantially from growth in capacity-related NFMs (e.g., square footage of facilities). Likewise, Dechow et al. (2011) observe that asset growth for fraud firms differs from growth in their number of employees. Both of these studies find that the relations between growth in financial data and related NFMs are closer for companies not committing fraud (i.e., percentage growths for both are more similar). Given these fraud firms received unqualified audit opinions, these studies raise the possibility that auditor attention to NFMs is currently insufficient to detect inconsistencies. From a practical perspective, auditors may not sufficiently rely on NFMs to verify financial information because of: (1) lack of easy availability, (2) budgetary pressures, (3) over-reliance on prior year workpapers that do not include analyses of NFMs, and (4) a lack of understanding about the NFMs that drive company performance (see Erickson et al. 2000; Brazel et al. 2004). We confirmed these conjectures with a sample of audit seniors (n=89). 3 We asked these auditors to note the reasons why they do not, at times, use NFMs. Consistent with the above, the most common reasons cited were: (1) lack of easy availability (58 percent), (2) lack of understanding about how NFMs drive company performance (29 percent), and (3) prior year workpapers do not include analyses of NFMs (18 percent). Auditors, at times, appear to not rely on NFMs for a variety of reasons, and the importance of these reasons is likely moderated by the specific experiences of the auditor (e.g., training, if their superiors endorse NFM use, etc.). 3 This sample of auditors is separate from the sample of auditors used to test this study s hypotheses. 10

13 Because prior empirical research suggests that auditors may not frequently rely on NFMs, we posit that auditor NFM reliance will not vary depending on whether inconsistencies exist or do not exist between financial measures and related NFMs. We therefore propose the following hypothesis in the null form: H1: When conducting substantive analytical procedures, auditors will not demonstrate a significant difference in their reliance on NFMs when NFMs are consistent versus inconsistent with financial results. Using a prompt to improve auditor reactions to inconsistencies Trompeter and Wright (2010) suggest that a structured tool that incorporates NFM data could provide valuable assistance to auditors when performing analytical procedures. Comparing revenue growth to NFM growth can be an efficient technique that effectively discriminates fraud firms from non-fraud firms (Brazel et al. 2009). Prior research in psychology suggests that such a technique might prompt auditors to be more attentive to NFMs when developing expectations for financial data (e.g., Petty and Cacioppo 1986). Prior audit studies have illustrated that prompts can improve audit effectiveness in the area of analytical procedures. For example, Bierstaker et al. (1999) find that a prompt can induce auditors to use more productive problem representations and identify seeded errors in an analytical procedures task. Likewise, Glover et al. (2005) observe that a prompt can activate knowledge bases that sensitize auditors to the weaknesses of an unreliable substantive analytical procedure. The prompt developed for this study has the potential to mitigate several of the previously noted cognitive and economic impediments associated with auditor NFM usage (see the Method section for additional details). First, the NFMs employed by our prompt, as well as those analyzed by Brazel et al. (2009), are easy for auditors to obtain and verify (e.g., employee headcount from human resource records). Second, regardless of whether the audit engagement 11

14 included the use of NFMs in the past, our prompt directs auditors attention to the NFMs that should correlate with financial performance. Consequently, the tendency to anchor on the same data sources used in the prior year should be reduced. Third, for auditors who do not know how to incorporate NFMs into their expectations, our prompt illustrates a simple and intuitive approach based on the method developed by Brazel et al. (2009). Last, the NFM analysis instigated by our prompt can be applied efficiently, which avoids undue strains on audit time budgets. The saliency of the NFM prompt The Heuristic-Systematic Model (HSM) suggests that the contextual features of a judgment affect how an individual processes information (Chen and Chaiken 1999). HSM predicts two modes of information processing by which individuals make judgments (Chaiken 1980, 1987; Chaiken et al. 1989; Chen and Chaiken 1999). Systematic processing involves the analytical and comprehensive treatment of judgment-relevant information. Heuristic processing involves the activation of judgmental rules (or "heuristics") that help an individual process cues more easily. However, heuristic processing can also lead to biased judgments (Chen and Chaiken 1999). While we expect our prompt will cause auditors to react to inconsistencies between financial data and NFMs, HSM theory would suggest the saliency of the prompt and the magnitude of the auditor s reaction depends on the level of risk associated with the audit engagement. Under HSM, more systematic processing is expected when an individual places greater emphasis on judgment effectiveness. Thus, the information processing that occurs under a high fraud risk assessment should lead auditors to systematically review the evidence sources available, more thoroughly analyze the inconsistent evidence presented by the prompt (and the 12

15 account balance expectation it develops), and provide more effective judgments (Brazel et al. 2004). When fraud risk is lower, auditor concerns for efficiency and the susceptibility to biases will likely rise (Brazel et al. 2004). Although auditor NFM use appears to be growing, auditors still lean toward other information sources (Hirst and Koonce 1996; Trompeter and Wright 2010). For efficiency purposes, auditors might be inclined to disregard the NFM data (even after the prompt) and instead rely on the familiar financial information sources that confirm the account balance (e.g., prior year balances, industry averages). Indeed, Glover et al. (2005) posit and find that auditors are biased towards relying on evidence that leads to expectations consistent with companies financial data. The existence of such a bias toward confirming evidence is well documented in the psychology and accounting literatures (e.g., Snyder and Swan 1978; Schustack and Sternberg 1981; Kida 1984; Cloyd and Spilker 1999; Kadous et al. 2003). Accordingly, we test the following hypothesis: H2: The positive effect of an NFM prompt on the extent of auditor NFM reliance will be stronger when fraud risk is high than when fraud risk is low. The auditor s expectation Our examination of auditor NFM reliance in H2 will describe the information sources used by auditors in our context. To determine whether the prompt and the fraud risk assessment interact to affect auditors in a way that has meaningful audit implications, we examine the account balance expectations developed by auditors. The company s NFMs in our experimental setting suggest a substantial decline in operations and revenues, and all other information sources support the client s current year revenue growth. Thus, as the effect of the prompt increases, we would expect lower expectations for revenue. We expect the level of fraud risk to moderate this 13

16 relation. We therefore test, in a setting where a client s financial performance outpaces growth in its related NFMs, the following hypothesis: Participants H3: The negative effect of the NFM prompt on auditor account balance expectations will be stronger when fraud risk is high (vs. low), and this relation will be mediated by the extent of auditor NFM reliance. METHODS Participants for Experiments 1 and 2 of this study were audit in-charge seniors because these professionals tend to perform the majority of analytical procedures on audit engagements (Prawitt 1995; Hirst and Koonce 1996; Trompeter and Wright 2010). Auditors completed either Experiment 1 or 2. A group of 39 (71) auditors completed Experiment 1 (2) at firm training sessions designed for in-charge seniors (i.e., all participants were audit seniors). Two of the authors were present at each of the sessions to observe the participants completing the experimental materials. Demographic data for our study s participants are provided in Table 1. The first (last) two columns relate to participants in Experiment 1 (2) Insert Table 1 here 4 Tests for differences in means between the first and second and between the third and fourth columns of Table 1 do not yield any p-values that are significant at p <.10. Thus, the randomization of participants to conditions appears to have been effective. Importantly, we observe no significant differences for manufacturing and substantive analytical procedures experience (i.e., experience in our setting and task), nor do we find a significant difference between conditions in participants problem solving ability (we use the same problem solving questions / measures as Bonner and Lewis (1990)) or in the usage of NFMs in practice. Bonner and Lewis (1990) find that general experiences, taskspecific experiences, and problem solving ability explain auditor performance / expertise. Thus, in our context, it is important that such characteristics are not significantly different across experimental conditions. Finally, nontabulated results indicate that the amount of time participants took to complete Experiment 2 (overall average of 64.7 minutes) is not significantly different across fraud risk conditions. 5 To obtain access to the participants at firm trainings sessions, the experimental materials used in this study were thoroughly reviewed by partners from the firm to assure that the materials were realistic and the task was reasonable for participants. After revising our materials as a result of this review, the experimental instruments described in this study were approved by the firm. 6 Differences in experience with substantive analytical procedures and in tendency to use NFMs in practice are not significant across experimental condition (see Table 1). Consistent with this, when we add experience with substantive analytical procedures as a covariate to the ANOVA model, the key interaction in Table 3 (prompt by fraud risk) is still significant at p=.0365, one-tailed. When we add NFM usage as a covariate, the key interaction in Table 3 (prompt by fraud risk) is significant at p=.0565, one-tailed. 14

17 Experiment 1: task, independent variables, and dependent variables Participants in Experiment 1 were provided with a case that placed them in the role of incharge senior of a hypothetical audit engagement of a publicly traded manufacturing company. The case materials were adapted from Brazel and Agoglia (2007) and included background information on the client along with audited prior year and unaudited current year financial statements. Participants were informed that their main task was to develop an expectation for the company s sporting goods revenue account (one of the company s 10 revenue accounts). Consistent with audit guidance and prior research (AICPA 1988; Trompeter and Wright 2010), all participants were then provided with data from the following sources to develop an expectation for the account: prior year financial data, budgeted amounts, regression analyses based on current financial information, industry data, and NFM information. Participants in Experiment 1 were randomly assigned to one of two conditions: (1) NFM growth was relatively consistent with revenue growth or (2) NFM growth was inconsistent with (i.e., lower than) revenue growth. For all participants, sporting goods revenue growth for the current year was approximately 9 percent. We rely on the descriptive data of Brazel et al. (2009) to manipulate the consistency of the revenue to NFM relation in a realistic manner. We provided participants with the following prior year and current year NFM data specifically related to the company s sporting goods revenue: patent information, number of customer accounts, square footage of production space, number of employees, and number of products. Brazel et al. (2009) find that, for non-fraud firms, these NFMs are highly correlated with revenue. 7 Participants receiving the consistent NFM information were provided with NFM growth of, on average, -1 7 If the NFMs used in this study were not diagnostic with respect to revenue, Brazel et al. (2009) would likely not have observed consistency between revenues and similar NFMs in their non-fraud sample. Using similar NFMs (e.g., employees), Dechow et al. (2011) also observe consistency in a non-fraud sample. If auditors in Experiment 1 do not demonstrate a significant difference in their reliance on NFMs when NFMs are consistent or inconsistent with financial results (test of H1), then it is likely that they are unaware of the potential diagnosticity of NFMs. 15

18 percent across all NFMs (10 percentage points different from revenue growth). 8 For participants in the inconsistency treatment, NFM growth was, on average, -19 percent (28 percentage points different from revenue growth). For all participants, all other data sources (e.g., prior year account balance, industry revenue growth) were consistent with the client s sporting goods revenue account. 9 After receiving the data, we asked participants to document their expectation for the client s revenue account. We used these calculations to determine which sources participants used to develop their expectation. Similar to the methods employed by Tan (1995) and Brazel et al. (2004), a researcher and an assistant (blind to experimental conditions) independently coded the sources that participants used to develop expectations (based on the inputs used / listed in the calculations provided by participants). For example, the coders each created an indicator variable set to 1 if the participant used prior year data to develop their expectation (coded 0 otherwise). Table 2 summarizes these data. All Cohen s (1960) kappa measures of agreement between coders for all the sources used in both Experiment 1 and 2 were above.700 and significant at p- value <.001. The small number of differences between coders was subsequently reconciled. We account for NFMs being one of several possible sources used by participants with the variable we label NFM Ratio. The NFM Ratio variable calculates an auditor s use of NFMs as a total of all sources used to develop their expectation (i.e., a measure of auditor NFM reliance). 8 Brazel et al. (2009) find some noise present between growth in financial data and growth in related NFMs for nonfraud firms (a mean difference of approximately 10%). Our experimental manipulation reflects this reality. Still, the 10% difference in the consistent condition may have been perceived as large by participants and could have muted the effect our manipulation in Experiment 1. 9 To determine the extent to which managers and partners perceive such an inconsistency as a fraud red flag, we surveyed 23 audit managers and partners (mean experience = 16.5 years) and asked the following: Please rate the effect of revenue growth outpacing NFM growth by 25 percent on the overall assessment of fraud risk. These professionals responded on an 11-point scale where 1 = definitely reduce fraud risk and 11 = definitely increase fraud risk. The mean response was These partners and managers rated the inconsistency as significantly higher than several other common red flags (e.g., high accruals, CFO turnover in the current year, management being extremely reluctant to post any audit adjustments (p-values <.05)). In sum, it appears that the inconsistency investigated in this study is deemed a high fraud risk and that experienced auditors would likely want this red flag conveyed to them by their audit seniors in the field. 16

19 If a participant did not use NFMs, the ratio is zero regardless of how many other sources were used. If a participant did use NFMs, the numerator will be one, and the denominator will range from one to five, depending on how many of the five available sources were used. For example, if a participant used only NFMs, the NFM Ratio measure would be 100 percent (1/1), if a participant used NFMs and prior year financial data, the NFM Ratio would be 50 percent (1/2), and so on. The NFM ratio and expectations provided by participants are the two key dependent variables examined in this study. Appendix A provides examples of how participants developed expectations and the coding of their documentation for the purposes of measuring the NFM Ratio. Experiment 2: task, independent variables, and dependent variables The task in Experiment 2 was identical to that of Experiment 1, except as follows. First, participants were randomly assigned to either a low or high fraud risk manipulation adapted from Wilks and Zimbelman (2004). Consistent with Wilks and Zimbelman (2004), the manipulation included an overall fraud risk assessment for the client, information about the management team, and a supporting checklist where fraud risks were designated as present or not present. 10 Second, all participants were provided with NFMs that were inconsistent with the revenue account as 10 While prior research tends to manipulate fraud risk or fraud cues as either low or high (e.g., Wilks and Zimbelman 2004), Brazel et al. (2010) find contemporary evidence that fraud risk assessments tend to be, on average, medium. As such, we manipulated fraud risk at three levels for this study: low, medium, and high. The fraud risk manipulation appears to have been effective. Participants responded to a manipulation check regarding the audit team s fraud risk assessment on a ten-point scale anchored by low, medium, and high. The means for the low, medium, and high fraud risk groups were 2.61 (sd = 1.36), 4.17 (sd = 1.70), and 7.48 (sd = 1.19), respectively. While a Bonferroni multiple-comparison test indicates that all three means are significantly different from each other, the difference between the means of the low and medium fraud risk groups (1.56) is substantially smaller than the difference between the means of the medium and high groups (3.31). We also find no significant differences in terms of NFM reliance and expectation development between our low and medium fraud risk groups. We therefore collapse these two experimental groups and, for the sake of simplicity, refer to them as the low fraud risk group. We obtain qualitatively similar results if we exclude the medium fraud risk group from our analyses and simply compare the low and high groups. Effects related to Hypotheses 2 and 3 are marginally significant when we utilize the three levels of fraud risk (p-values =.081 and.098, respectively). A lack of auditor reaction to fraud risk varying from low to medium complements, and is consistent with, earlier fraud research illustrating auditors not modifying the nature of their planned testing in response to variation in fraud risk (e.g., Glover et al. 2003; Asare and Wright 2004). 17

20 described above. Third, after documenting and providing an initial expectation for the current year revenue account (from which we obtained our pre-prompt measures of NFM ratio and expectation), all participants received an NFM prompt (see Appendix B). Similar to Brazel et al. (2009), the prompt asked them to consider and develop an additional expectation that combined the prior year revenue account ($41,508,009) and the current decline in NFM growth (-19 percent). Therefore, like the prompt used by Glover et al. (2005) and how a firm might practically incorporate such a forensic procedure into their audit approach, our NFM prompt was within-participants manipulation. Consistent with the analyses of Brazel et al. (2009), all NFMs were equally weighted by the prompt. Based on the tolerable threshold provided to all participants ($4,410,817), an expectation based solely on the prompt ($33,357,513) is clearly materially different from the client s current year balance for sporting goods revenue ($45,372,585). Participants were then asked to develop and document their final expectation for the revenue account (from which we obtained our postprompt measures of NFM ratio and expectation). They were informed that they were free to form their final expectation however they best saw fit, including using their initial expectation, their expectation developed from the prompt, or any combination of their previous expectations. RESULTS Experiment 1 When conducting substantive analytical procedures, H1 states that auditors will not demonstrate a significant difference in their reliance on NFMs when NFMs are consistent versus inconsistent with financial results. If auditors do react to inconsistencies between financial and nonfinancial measures, in the NFMs Inconsistent condition we should observe (1) significantly higher NFM ratios, (2) lower expectations, and (3) a higher percentage of auditors indicating a 18

21 material difference existed between their expectation and client s revenue balance (vs. the Consistent condition). These measures of auditor reactions in both the NFMs Consistent and NFMs Inconsistent conditions are presented in Table 2. The first two rows in Table 2 provide data related to Experiment 1. Consistent with H1, all three measures are not significant between conditions (non-tabulated p-values are.42,.54, and.96, respectively). Particularly, as noted in Table 2 (second row, third column), only 1 of the 21 participants in Experiment 1 s Inconsistent condition noted a material difference between his or her expectation and the account balance. These results suggest that auditors, at least in the absence of a prompt, are unlikely to react to red flag inconsistencies between financial measures and related NFMs. Insert Table 2 here Table 2 also provides descriptive data on the variety of sources auditors used in Experiment 1. These data indicate that, consistent with prior research and other anecdotal evidence, auditors tend to focus on prior year financial data when generating expectations for current year account balances. Regardless of whether NFMs were consistent or inconsistent with client revenues, over 60 percent of auditors relied on prior year financial data (e.g., the prior year account balance) when developing their expectation. Approximately half of the auditors made use of the regression data provided in the case, and just over 40 percent incorporated industry data into their expectation. NFMs were the fourth most widely used information source, with just over a third of auditors utilizing NFMs. Only client budget data was used less than NFMs, with just over ten percent of auditors using it. 11 Experiment 2 11 Post-experimentally, we asked participants in Experiment 1 the following: In practice, when performing substantive analytical procedures... what percent of the time do you use NON-FINANCIAL MEASURES... as the primary basis for your expectations for current balances? The mean response was 35.5% and the difference in NFM usage is not different across condition of Experiment 1 (40% and 31%, p-value =.351). This provides some evidence that auditor use of NFMs in our experimental settings likely reflects their use of NFMs in practice. 19

22 Hypothesis 2 Our second hypothesis predicts an interactive effect between the prompt and the fraud risk assessment on the NFM Ratio. Specifically, the positive effect of the prompt on the NFM Ratio will be stronger when fraud risk is high (vs. low). Table 3 presents the results from an ANOVA with NFM Ratio as the dependent variable and Prompt and Fraud Risk as within- and between-participants manipulations, respectively. Insert Table 3 here Panel A of Table 3 indicates that the main effect for the fraud risk manipulation is not significant (p-value =.27). Thus, our participants did not simply follow a risk-based approach or more deeply consider the disconfirming evidence provided by the NFMs under higher fraud risk. We find the main effect for the NFM prompt to be highly significant at p-value <.001, with the prompt increasing auditors reliance on NFMs (13.7 percent for the initial expectation and 29.6 percent for the final expectation, respectively; see Panel B). Of particular relevance to H2, the NFM Prompt by Fraud Risk interaction is significant at p-value <.05. After being prompted to consider the revenue / NFM relation, auditors increased their reliance on NFMs to a greater extent when fraud risk was high (vs. low), as illustrated in Panel B. These results are consistent with H2, indicating that the positive effect of the prompt on NFM reliance is stronger when fraud risk is high. Hypothesis 3 H3 predicts and Figure 1 illustrates that the interactive effect of the NFM prompt and the fraud risk assessment on auditor account balance expectations will be mediated by the extent of auditor NFM reliance (i.e., the NFM Ratio). To test H3, we conduct a mediation analysis (see Baron and Kenny 1986; Wilks and Zimbelman 2004; Brazel and Agoglia 2007). 20

23 Insert Figure 1 here Statistical evidence of NFM reliance mediating the relation between the NFM Prompt by Fraud Risk interaction on account balance expectations first requires that the interaction significantly affect the Expectation variable (see Table 4, Panels A1 and A2). This analysis, similar to our tests of H2, indicates that the main effect of Fraud Risk is not significant (p-value =.24; see Panel A1). The main effect of the NFM Prompt in Table 4 is significant at p-value <.001 and in the expected negative direction (mean pre- and post-prompt expectations were $44,570,040 and 40,925,340, respectively; see Panel A2). 12 Of particular relevance to H3 is that the NFM Prompt and Fraud Risk significantly interact, such that the negative effect of Prompt on Expectation is greater when Fraud Risk is high versus low (p-value =.037). Insert Table 4 here It is important to note that the prompt / high fraud risk cell is the only condition in which the mean expectation drops below the significant difference threshold, which would trigger further investigation (see graph in Table 4, Panel A2). This result suggests that the prompt / fraud risk interaction has effects on auditors expectations that are not only statistically but also practically significant. In short, as prescribed by SAS No. 56 (AICPA 1988), auditors in the high fraud risk condition in our experiment should question and further investigate the difference between their expectation and the reported revenue balance. Auditors in the low fraud risk condition, the other hand, likely would forego further investigation (see footnote 2). Identifying and investigating inconsistencies between financial data and related NFMs is a necessary first 12 The mean for the post-prompt expectation ($40,925,340) was substantially different from the expectation derived directly from the prompt ($33,357,513). This suggests that participants in Experiment 2 were not simply following instructions or assumed that it was mandatory that the results from the prompt be incorporated into their final expectation. 21