An Experimental Study of Internal Control Judgments ROBERT H. ASHTON*

Transcription

1 Research Reports An Experimental Study of Internal Control Judgments ROBERT H. ASHTON* The work of independent auditors involves two basic activities evidence collection and evidence evaluation. Substantial agreement exists concerning the types of evidence that should be collected in particular situations and the appropriate collection techniques. The evidence-evaluation activity, however, is less susceptible to codification. As a result, the auditing profession relies heavily upon "professional judgment" in evaluating audit evidence. For example, the AICPA has stated that "most of the independent auditor's work in formulating his opinion on financial statements consists of obtaining and examining evidential matter. The measure of the validity of such evidence for audit purposes lies in the judgment of the auditor." ^ The importance of judgment in auditing is accepted almost without question; yet auditors' judgments are seldom subjected to systematic research. A typical rationale for this situation is the AICPA's statement that "judgment is the most Important factor in the making of any audit, but in many situations it is practically impossible to write out in specific language how the auditor applies judgnient." ^ The study reported in this paper was concerned with the evidence-evaluation function of independent auditing. One type of audit evidence was * Assistant Professor, University of Texas at Austin. This paper is based upon the author's doctoral dissertation; for the full report, see R. H. Ashton, Judgment Formation in the Evaluation of Internal Control: An Application of Brunswik's Lens Model (Minneapolis: University of Minnesota, 973). ' Committee on Auditing Procedure, Auditing Standards and Procedures, Statements on Auditing Procedure, no. 33 (New York: AICPA, 963), p. 34. " Committee on Auditing Procedure, A Case Study of the Extent of Audit Samples (New York: AICPA, 955), p

2 44 JOURNAL OF ACCOUNTING RESEARCH, SPRING, 974 considered ^internal control. Auditors were asked to judge the strength of an internal control subsystem (payroll) as a function of various combinations of internal control indicators. Specifically, two major aspects of internal control judgments were studied: () consistency of judgments, and () the perceived importance of the various indicators to the judgments of particular auditors. Of course, these two aspects of judgment are related since the degree of judgment consistency among auditors is influenced by their agreement concerning the perceived importance of the various indicators. The remainder of the paper presents () a brief discussion of the importance of studying these aspects of audit judgment, () a description of the experimental task and the experimental design, (3) the results of the experiment, and (4) a discussion of possible implications of the research. Judgment in Internal Control Evaluation The requirement to review and evaluate internal control is evidenced by the second standard of fieldwork: "There is to be a proper study and evaluation of the existing internal control as a basis for reliance thereon and for the determination of the resultant extent of the tests to which auditing procedures are to be restricted." ' The primary purpose of the internal control review is to enable the auditor to determine the particular auditing procedures to be applied, the timing of those procedures, and the extent of their application. JUDGMENT CONSISTENCY In order to highlight certain points concerning the consistency (or inconsistency) of internal control judgments, it is useful to consider four steps in the audit process where judgment is required:. Evaluation of internal control;. Determination and performance of substantive tests; 3. Assessment of adherence to generally accepted accounting principles and consistent application thereof; 4. Integration of all previous judgments and evidence into determination of the type of audit opinion to be issued. The first two steps primarily involve individual judgment processes (although such judgments may be reviewed by superiors), and the other two steps primarily involve group judgment processes. At least two kinds of research questions concerning audit judgment inconsistency are apparent: () research concerning extent of inconsistency, and () research concerning effects of inconsistency. The research reported here was concerned with the former. This issue was examined in the context of an experiment involving the evaluation of a relatively simple hypothetical internal control subsystem. If auditors' judgments are found to be very ' Auditing Standards and Procedures, p. 6.

3 INTERNAL CONTROL JUDGMENTS 45 inconsistent in this simple situation, then audit judgment inconsistency, and its effects, deserve further study. Since the experiment focused upon internal control evaluation, it was concerned with the judgment processes of individual auditors rather than with group judgment processes. Before describing the experiment in detail, it is important to note some possible effects of inconsistent internal control judgments. It should be emphasized, first, that no empirical or experimental studies have dealt with the effects of inconsistent audit judgments in any of the four areas mentioned. My concern for this topic, as well as the concern of other writers, is based on an intuitive feeling for the situation. Consider, for example, the assumption that "the learned statement user can distinguish between the different grades [of opinions]," * and that this may lead to the spectilation that "a financial statement user, in making his resource allocation decision, places less reliance on the financial statements in correspondence to the degree to which the audit report is qualified." ^ If these assumptions are correct, it follows that inconsistent judgments of internal control strength can affect users' resource allocation decisions, since the formulation of the audit opinion is based, in part, on the evaluation of internal control. The effects of inconsistent internal control judgments may also be reflected in the cost and/or quality of an audit. That is, the audit is conditioned by the auditor's judgment of the strength of the internal control system, regardless of the controls actually employed or the evidence gathered to evaluate them. Variations in judgment by the same auditor at different points in time, or by different auditors at the same point in time, may cause the cost and/or quality of an audit to fluctuate. Of course, this assumption has also not been tested empirically or experimentally. A major purpose of this study was simply to determine the extent of judgment inconsistency in the evaluation of a hypothetical internal control situation. The study did not attempt to assess the effects of judgment inconsistency. DESCRIPTIVE MODELS OF JUDGMENT A descriptive model of each auditor's judgments was constructed in order to provide a partial explanation for the extent of inconsistent internal control judgments which might be observed. These models are descriptive representations of the weightings given to the various internal control indicators by individual auditors. Obviously, the more discrepant the weightings of auditors, the more inconsistent will be their internal control judgments. Differential weighting of internal control indicators is a topic which has appeared previously in the auditing literature but on which no descriptive «H. M. Anderson, J. W. Giese, and J. Booker, "Some Propositions about Auditing," The Accounting Review (July 970), p. 65. * D. R. Carmichael, The Auditor's Reporting Obligation, Auditing Research Monograph no. (New York: AICPA, 97), p..

4 46 ROBERT H. ASHTON research has been conducted. Brown pointed out that "several auditors might judge the effectiveness of a given system of internal control quite differently.... This condition develops primarily from the use of different methods of appraisal, but can also arise because auditors place different emphasis on the relative importance of various factors of internal control." ' The descriptive models will provide some insights on whether different auditors base their evaluations on different internal control indicators or consider some indicators more important than others. It should be noted that the effects of inconsistent internal control judgments might be negated by other factors. In a hke manner, the effects of differential weightings of internal control indicators might also be negated. In the study reported here, one objective was to identify the descriptive models which represented the auditors' judgments. The Experiment The experiment was based upon a judgment research technique developed by psychologists." The technique, which involves the representation of individual judgment "strategies" via analysis-of-variance, is applicable in situations in which a judgment must be made on the basis of numerous pieces of information which may be differentially relevant to that judgment. In addition, the human judge may not know either the extent or the functional form of the relationship between an item of information and the "optimal" judgment. These characteristics are present in situations in which auditors evaluate (judge) the effectiveness of a set of internal controls. Descriptive analysis-of-variance models obviously do not constitute the only technique for representing judgments. In this particular case, it was believed to be a good compromise between the simplistic approach of asking the auditors to weight the various internal control indicators and the very complex approach used by Heinmuntz* and by Clarkson.' These two researchers had experts "think aloud" into a tape recorder as they made complex judgments. Then they constructed computer programs which simulated the experts' thought processes. The resulting programs were sequential representations of the experts' verbal reports. The research reported in the present paper was concerned with descriptive models of the «R. G. Brown, "Objective Internal Control Evaluation," The Journal of Accountancy (November 96), p. 50. ' P. J. Hoffman, P. Slovic, and L. G. Rorer, "An Analysis-of-Variance Model for the Assessment of Configural Cue-Utilization in Clinical Judgment," Psychological Bulletin (May 968), pp * B. Kleinmuntz, "The Processing of Clinical Information by Man and Machine," in Formal Representation of Human Judgment, ed. B. Kleinmuntz (New York: John Wiley & Sons, 968), pp ' G. P. E. Clarkson, Portfolio Selection: A Simulation of Trust Investment (Englewood Cliffs, N.J.: Prentice-Hall, 96).

5 INTERNAL CONTROL JUDGMENTS 47 weightings attached to internal control indicators and was not concerned with sequential thought processes used by the auditors.^" THE EXPERIMENTAL TASK In the evaluation of a client's system of internal controls, auditors ts^pically concentrate upon individual internal control "subsystems," for example, cash receipts, inventories, etc. The payroll subsystem was chosen for this research study for two reasons. First, the review of the payroll subsystem is important because of the factor of relative risk. In fact, the review of internal control may be the most important aspect of the payroll audit. Second, the characteristics of adequate payroll internal control should be familiar to all auditors. The subjects taking part in the research study were given a brief narrative describing a hypothetical manufacturing company. Included in this narrative was a description of some basic structural aspects of the company's payroll internal control subsystem. In order to keep the subjects' task manageable, six indicators of payrou internal control were selected. They were presented to the subjects in the form of questions which might appear on an internal control questionnaire. Each question was preanswered "yes" or "no." A "no" answer always represented a weakness in the internal control subsystem. In conjunction with the unchanging background information, each subject made thirty-two judgments of internal control strength as a function of thirty-two different combinations of "yes" and "no" answers to the six questions. Their judgments were made on a scale from (extremely weak) to 6 (adequate to strong).'^ An example of one combination of answers to the six questions (hereafter called a "case") is presented in table. The first four questions are based upon three characteristics of soimd internal control discussed in Statements on Auditing Procedure, no. 33."^^ " For a discussion of descriptive judgment models in an accounting context, see N. J. Gonedes, "Remarks on 'Empirical Research in Accounting : An Appraisal'," in Accounting Research : A Critical Evaluation, ed. N. Dopuch and L. Revsine (Urbana: University of Illinois, 973), p. 8. ' An eight-point rating scale was used in a pilot study. The points were labeled as follows: extremely weak; very weak; 3 weak; 4r slightly weak; 5 adequate; 6 ^moderately strong; 7 very strong; 8 extremely strong. The pilot study participants hardly used points 6, 7, and 8. Their comments indicated that () if anything was wrong with the payroll internal control subsystem, then its rating should descend very rapidly from the upper end of the scale, and () the auditing procedures applied in a particular case would probably not be significantly affected by exceptional internal control strength. Therefore, the upper end of the original eight-point scale was discarded, and no further attempt to distinguish between "adequate" and various degrees of "strong" internal control was made. ^ The three characteristics involve the "plan of organization," the "system of authorization and record procedures," and "sound practices." A fourth characteristic, "quality of personnel," was not explicitly incorporated into this research. After the data for this experiment had been collected, the AICPA Committee on Auditing

6 48 ROBERT H. ASHTON The fifth question concerns the work of internal auditors. Generally, a competent and efficient internal audit department is assumed to be a valuable contribution to internal control, so the independent auditors may reduce the extent of their detailed tests when such a department exists.^' The sixth question concerns the results of the internal control examination of the previous year. Auditors may use past results as a starting point for the examination of the current year, since the past is often the best indicator of present conditions. Moreover, past results are important in guiding the auditor's current review of an internal control subsystem. Instead of a complete review, only a few factors need to be examined each year. That new information is then combined with the results of previous examinations in evaluating the effectiveness of the controls. Notice that the questions used in this experiment may take on only dichotomous, as opposed to continuous, values, that is, "yes" and "no." This is representative of actual practice in which internal control questionnaires prepared and administered by pubkc accounting firms must be answered in that manner." THE EXPERIMENTAL DESIGN The thirty-two combinations of "yes" and "no" answers were chosen according to a one-half fractional replication of a ' factorial design.'^ The six questions constituted the/actors in the design; the answers to the questions constituted levels of the factors; the auditors' ratings constituted the dependent variables; and the data were analyzed separately for each auditor. In this context, a significant main effect" for a particular question im- Procedure published a report which replaced this fourfold classification of internal control characteristics with a sixfold classification. The differences between the two classifications are ones of form, not substance. See Committee on Auditing Procedure, The Auditor's Study and Evaluation of Internal Control, Statement on Auditing Procedure, no. 54 (New York: AICPA, 97). '^ R. K. Mautz, Fundamentals of Auditing (New York: John Wiley & Sons, 964), p * Internal control questionnaires often include three possible answers to each question: "yes," "no," and "not applicable." Therefore, the answers of "yes" and "no" may not represent the entire distribution of levels in practice. This should not present a problem in the experiment, however, since only six independent variables were presented to the subjects, and all were presumed to be applicable to the case. " W. G. Cochran and G. M. Cox, Experimental Designs (New York: John Wiley & Sons, 957), pp ; O. Kempthorne, The Design and Analysis of Experiments (New York: John Wiley & Sons, 95), pp '^ In fractional factorial designs, a comparison which estimates any effect also estimates one or more other effects; the effects, which are said to be aliases of one another, cannot be independently estimated. In the design used here, main effects are aliased withfive-factorinteractions, and two-factor interactions are aliased with four-factor interactions. Significantly, evidence exists which indicates that such higher-order interactions in studies of human judgment are negligible. See, for example, L. R. Goldberg, "Sim.ple Models or Simple Processes? Some Research on Clinical Judgments," American Psychologist (July 968), p. 49; P. J. Hoffman, "Cue-

7 INTERNAL CONTROL JUDGMENTS 49 plies that the auditor's responses varied systematically with the level of that particular question. In other words, it made a difference to the auditor whether the question was answered "yes" or "no." A significant two-factor interaction" implies that the auditor was responding to particular patterns of answers to the questions, that is, the effect of variation of one question upon his responses differed as a function of the level of another question.'* In the present study, it would be surprising to find large interactions because a "no" answer to every question implies a weakness in the internal control subsystem. We would not expect different patterns of answers to reverse the implications of particular cues. A similar situation was described by Hoffman et al. in a medical diagnosis study:... strong configural or interactive effects will be found only when the implication of a particular cue is radically different when associated with one pattern of other cues, rather than with another. In particular, the presence of Sign X might imply malignancy when Signs A, B, and C are present, whereas the absence of Sign X might imply malignancy when Signs A, B, and C are absent. For interactive effects to account for a substantial portion of the total predictable variance, such reversals would have to be the rule, rather than the exception. This seems highly unlikely...within the framework of most tasks familiar to human experience. What is more likely is that certain patterns augment or attenuate the importance of particular signs without reversing the implication." Where small Iateractive effects were identified in the present study, it is more likely that particular patterns of answers had augmented or attenuated, rather than reversed, the implications of particular questions.^" Perhaps it is to be expected that the first two questions would, on the average, account for more variance in the auditors' judgments than would other questions. Both questions deal with separation of duties, an internal control characteristic which is especially important in the case of payroll. However, it is more important to note the differences in the extent to which Consistency and Configurality in Human Judgment," in Formal RepresertiaMon of Human Judgment, p. 73; and P. Blovic, D. Fleissner, and W. S. Bauman, "Analyzing the Use of Information in Investment Decision Making: A Methodological Proposal," Journal of Business (April 97), p. 96. " In this particular design, only main efifects and two-factor interactions are estimable. The three-factor interactions served to estimate the error term for the analysis. See, for example, Cochran and Cox, p. 8; and O. L. Davies, ed.. The Design and Analysis of Industrial Experiments (New York: Hafner Publishing Co., 966), pp * Hoflman et al., p. 34. " Ibid., p " In addition, it has been pointed out that the analysis-of-variance technique may be biased toward linearity, i.e., the attribution of judgment variance to main effects. See D. B. Yntema and W. S. Torgerson, "Man-Computer Cooperation in Decisions Requiring Common Sense," IRE Transactions on Human Factors in Electronics, HFE- (96), pp. 0-6; J. B. Sidowski and N. H. Anderson, "Judgments of City- Occupation Combinations," Psychonomic Science, March 5, 967, pp ; and B. F. Green, Jr., "Descriptions and Explanations: A Comment on Papers by Hofiman and Edwards," in Formal Representation of Human Judgment, pp

8 50 ROBERT H. ASHTON the various auditors relied upon these two questions, and the other four, in formulating their internal control judgments. Concentrating upon differences among the auditors' descriptive models enables us to understand better any inconsistency in their resultant judgments. SUBJECTS AND PROCEDURE The subjects were sixty-three practicing auditors who were employed by four public accounting firms in the Minneapolis/St. Paul area. The four firms consisted of two large national firms, a regional firm, and a local firm. A great majority of the auditors had two or three years of experience. From discussions with the auditors' superiors, it was determined that the typical amount of experience required before an auditor is allowed to make this type of judgment is two and one half to three years. The experiment was conducted by visiting the public accounting firms and meeting with the auditors in small groups. The participants were given a brief oral introduction to the judgment task and were then presented with the case materials. The auditors typically worked at the task for thirty to forty minutes. The second administration of the experiment, which was necessary in order to assess the consistency of judgment over time, was conducted in the same manner as thefirst.^^ The time interval between the two administrations ranged from six to thirteen weeks. On the second administration, the auditors again worked at the task for thirty to forty minutes. DATA ANALYSIS Two types of consistency were evaluated: () consensus, or consistency across auditors at the same point in time, and () stability, or consistency over time for the same auditor using the same data.^^ Consensus was evaluated by correlating the ratings (judgments) of each auditor on the thirtytwo cases with the ratings of each remaining auditor. Stability was evaluated by correlating the ratings made by each auditor on the first administration of the experiment with his own ratings from the second administration. For the purpose of constructing descriptive models of judgment, /''-ratios were computed for the six main effects and fifteen two-factor interactions from the data from each auditor. Then ca^ was computed for each main effect and two-factor interaction.^^ Here, w^ is an estimate of the proportion ''^ One exception occurred. In the case of the regionalfirm, it was necessary to conduct the second administration by mail. Five auditors did not return their experimental materials; therefore, only fifty-eight of the sixty-three auditors repeated the experiment. L. R. Goldberg and C. E. Werts, "The Reliability of Clinicians' Judgments: A Multitrait-Multimethod Approach," Journal of Consulting Psychology (June 966), p. 99. «W. L. Hays, Statistics (New York: Holt, Rinehart & Winston, 963), pp , 407.

9 INTERNAL CONTROL JUDGMENTS 5 of variance in judgment that is accounted for by the specific main effect or interaction. Therefore, w^ measures the extent to which each auditor utilized each of the sis internal control questions (and their interactions) in formulating his internal control judgments. Remits^* JUDGMENT CONSISTENCY Judgment Consensus. The average correlation between the ratings of all pairs of auditors on the thirty-two cases was approximately.70. The range of correlation coefficients was generally from the lower.40s to the higher.80s, but a small number of coefibicients were between.06 and.40 (about 3 percent) and between.90 and.93 (about percent). Average correlations were obtained between all auditors in the same firm and between all pairs of firms. The same data were obtained for the various experience levels. Again, the average correlation was approximately.70, indicating that, on the average, an auditor agreed with members of other firms (or experience levels) as well as he agreed with members of his own firm (or experience level). Of course, there was a range of agreement, both within firms (and experience levels) and between firms (and experience levels). Levels of agreement were related to differential weightings of the cues by the auditors. This is discussed at a later point. Judgment Stability. Recall that the consistency of an individual auditor's judgments over time was determined by conducting the experiment twice; the time interval between administrations ranged from forty-three to ninety-four days. The experiment was repeated by fifty-eight of the sixtythree auditors. Judgment stability was assessed by correlating the ratings of the thirtytwo cases made by an auditor on the two occasions. The coefibcients ranged from a low of.43 to a high of.96, with an average of.8. Only four of the fifty-eight coefficients were below.70. The coefficients differed very little across firms or experience levels. The average correlations representing firms or experience levels ranged from.78 to.85. In general, stability was greater than consensus, indicating greater agreement between an itidividual's judgment at two points in time than between different individuals' judgments at the same point in time, given the same data in all cases. DESCRIPTIVE MODELS OF JUDGMENT To obtain further information about judgment consistency, a descriptive analysis-of-variance model was constructed for each auditor. On the average, the total of the w^ values for the six main effects was just over 80 percent (the range was 48 percent to 96 percent). For the fifteen interactions, * All data reported are taken from the first administration of the experiment except, of course, the data pertaining to judgment stability. Data from the second adm^inistration demonstrated the same relationships as that from the first.

10 5 ROBERT H. ASHTON this total was 6.4 percent (the range was zero to 7 percent). The latter indicates that the auditors did not look for patterns of answers to the six questions. Instead, they evaluated the effect of each question independently of the effects of other questions in the same case. At least five of the six main effects were significant (at the.05 level) for two-thirds of the auditors, while the number of interactions reaching significance was generally zero or one, as indicated in table. More important than averages, however, are the models which describe the judgments of individual auditors. These models are presented, for all sixty-three auditors, in table 3. The numbers in the table refer to the values of c/ for particular questions (main effects) and combinations of questions (interactions). Only u>^ values for significant effects are presented.^^ Note the scarcity of significant interactive effects in table 3. Moreover, when an interaction is significant, it accounts for a very small proportion of variance in judgments, as indicated by co^ values. Table 3 also reveals the fact that some of the questions were considered much more important than others. Questions and account for 6. percent and 5. percent, respectively, of the variance in judgments when averaged across all auditors. Questions 3, 4, 5, and 6 account for only 6.7, 7., 8.5, and 6.6 percent, respectively. Intuitively, we would expect questions and to be very important in a payroll audit since both questions involve separation of duties.^^ Despite the overwhelming importance of questions and, individual differences existed among the auditors such that each question was considered mast important by at least one auditor, and each was considered least important by at least one auditor. Differences in the extent of utilization of the six questions are directly related to the extent of inconsistency in the overall judgments made by the auditors. Some examples concerning judgment consensus, or the extent of agreement among different auditors at the same point in time, are presented in table 4. These examples indicate that the judgments of two auditors who relied upon the same questions to approximately the same extent (e.g., auditors 4 and 44 or auditors and 4) were substantially in agreement. Likewise, auditors who relied upon different questions (e.g., auditors and 44) exhibited a lower level of agreement in their overall judgments. Similar relationships held in the case of judgment stability, or the extent of agreement between judgments of the same auditor at two points ia tithe. Those auditors who relied upon the same questions on both administrations of the experiment exhibited relatively high levels of staboity, while those auditors who changed their patterns of reliance exhibited lower levels of stability. " The significance level used is.05. The great majority of main effects were significant well beyond the.0 level, but almost no interactions were significant at that level. " Since these two questions always appeared as the first two questions in every case, we have no data either to support or deny the possibility of an order effect.

11 Summary and Implications INTERNAL CONTROL JUDGMENTS 53 This study has provided experimental evidence concerning the extent of judgment inconsistency in the evaluation of internal control. In general, the judgments of the sixty-three auditors studied were found to exhibit a fairly high level of consistency (both consensus and stability). But when specific auditors were considered, some inconsistency was found to exist in this relatively simple judgment task. Additional research is needed in order to determine why the judgments of particular auditors are inconsistent. For example, do individual factors, situational factors, or some combination of both account for judgment inconsistency? Research is also needed to determine the effects of inconsistent internal control judgments. If subsequent research provides evidence of deleterious effects of inconsistent internal control judgments, then reduction of such inconsistency laay be a worthy objective for public accounting firms to pursue. A prerequisite to the reduction of judgment inconsistency is the recognition of its existence and the identification of its possible reasons. The techniques demonstrated ia this paper can be used for both purposes. The correlation technique allows us to identify those auditors who disagree with each other, as well as those auditors whose judgments are relatively inconsistent over time. The analysis-of-variance technique provides a means for identifying possible reasons for those inconsistencies. The possible reasons would be represented by differential weightings of the various internal control indicators used. The knowledge gained from an application of these techniques could guide firms that wish to influence the extent to which auditors rely on particular indicators. Since criterion values are generally not available, that is, the "true" state of an internal control subsystem may not be known, it is not possible to derive "true" statistical weights toward which the auditors should strive. It is possible, however, for someone in a position of authority within a firm to prescribe approximate guidelines for statistical weights. Of course, the specification would be made on a relative basis and might, initially, involve only a ranking of the indicators. The firm might then undertake training programs to encourage auditors to use the various internal control indicators to the relative extent prescribed for each indicator. This would amount to the adoption of a judgment "strategy" to be promoted by the firm and communicated to the auditors who make such judgments. With the preferred strategy "out in the open," discussion among those involved might lead to changes in the strategy as originally specified. Different strategies would be necessary for different internal control subsystems. Likewise, the basic strategy might have to be adjusted for the same subsystem in clients of different sizes (or perhaps for each client), but such adjustments could be made on the basis of specific client differences, and an entirely new strategy would not have to be developed for each review.

12 54 ROBERT H. ASHTON A large number of studies exist which indicate that individuals can learn to adjust the extent to which they rely upon various indicators if they are given feedback concerning the validities of the indicators.^^ In the case of internal control judgments, the "validities" of the various indicators would be the weights established by the firm. One study indicated that a "native" group of subjects, that is, those with very little experience with the judgment task, was more successful than "middle" and "expert" groups in learning to use particular cues.^^ This indicates that the payoff from such a training program might be greater if begun at a relatively early point in the auditor's career with the firm. The purpose of these comments is not to suggest that professional audit judgment be eliminated or relegated to a secondary position vis-^-vis "mechanical" judgment models. A training program such as the one described above might be used simply to assist auditors in structuring their judgment processes. Of course, the suggestion that public accounting firms consider this type of training raises more questions than it answers. Ample room exists for subsequent research. For example, the specification of preferred weights by persons in positions of authority within public accounting firms could constitute a study in itself. Similarly, it might be interesting to compare the preferred weights set by different firms. Estimation of the costs of applying judgment analyses within a specific firm and the benefits of reduced judgment inconsistency are other areas for research. Some estimates of cost could be made in specific firms, but a quantification of benefits appears very difficult. However, since inconsistent internal control judgments may first affect the selection and extent of auditing procedures to be applied, a first step in the estimation of benefits could be research on the relationships between internal control evaluation and the audit program. In summary, this study was concerned with the evidence-evaluation activity performed by independent auditors. Some experimental evidence was provided concerning the extent of judgment inconsistency in the evaluation of one type of audit evidence. A method for reducing judgment inconsistency within public accounting firms was also outlined. It should be emphasized that the evidence presented here constitutes only a first step in the analysis of inconsistent audit judgments. The full practical implications of such a study must await future research on the effects of inconsistent audit judgments. "' For example, see C. R.. Peterson, K. R. Hammond, and D. A. Summers, "Multiple Probability Learning with Shifting Weights of Cues," American Journal of Psychology (December 965), pp ; and S. A. Summers," The Learning of Responses to Multiple Weighted Cues," Journal of Experimental Psychology (July 96), pp ^ L. G. Rorer and P. Slovic, "The Measurement of Changes in Judgmental Strategy" (Paper presented at the convention of the American Psychological Association, New York, 966).

13 INTERNAL CONTROL JUDGMENTS 55 TABLE Example of an Internal Control Case Used in the Experiment Case No Yes No. Are the tasks of both timekeeping and payment of employees adequately separated from the task of payroll preparation? X. Are the tasks of both payroll' preparation and payment of employees adequately separated from the task of payroll bank account reconciliation? 3. Are the names on the payroll checked periodically against the active employee file of the personnel department? 4. Are formal procedures established for changing names on the payroll, pay rates, and deductions? X 6. Is the payroll audited periodically by internal auditors? X 6. Was the internal control over payroll found to be satisfactory during the previous audit? X extremely VPTV wpak substantial some not quite adequate weak ^ weak weakness weakness adequate to strong X X TABLE Number of Main Effects and Interactions Reaching Significance'^ for the Auditors No. of significant effects Main effects Interactions No. of auditors for whom specific no. of main effects or interactions reached significance * At the.05 level of sxgmficance.

14 pa M ; toi 3 * Effect CO CO "^ CO C- CO OC C (M O^ (M Oi "^ (T^ t^ CO O N O O (M -< T-H OS OS O3 CO CO O O O IOtOCOCOCO(MCO<M (M ] 00 CO e^ J -^ N «O «5 rh I O O 5 S W S CO ^!:O CO O CO im lo OO lo O O O -H -^ O UD -^ O) "=J* O i US IM O CM CM lo O to O O!M g SS g S 3 3 o g o -M Ol i n CV3 CO!M CO cs 3 ^i 3 o ^ ^ sg M S S g S 3 * ^ w o o o «^ M O 8 O S I I I I I M g o o o g s g i I S I I IIII IIII gill I S I IO I r - O O C O C O t >. c O < N i! «T:fi l >. O O O O l O O l o CO O S a s - ^! H ] I M - * f [ " ^ O M ^ M t M O O o i o i o i o o l l o CO CO OO r-i to =O CO OS!>. "^ O C'Q -H O -H O 00 to OS OS W5 -* T-H CO O O O Ol CO m O3 T-i -H CO CO O CO O O O O 00 :^ O <M to tn CO CO IM i-h O TH CO C<l CO SO CO -H T-l CO t CO OS O TH rh C^ * CO cq io O O O O S? M M O O O O rh" 00 CO '^ OS CO CO -rfh O 00 CD CO iro CO O O S? ss M -H OS CO IM 00 "* o e^ OO o CO [ CO -tjf rm -H - OS I>. -*! C3a O -* (M O C^ (M T-H O OO CO CO CO t- t- (M l-l im T-i O O t- ^ t^ (M 00 j cq -H o <-H (M g :^ ^! o lo GO OS J j <*< OS (M -H OS -H (M t- O O O O CO Sq i S N I! IM jh g g S S g g 3 o S O "tji CO (M to 00 (M -H CO O I O --< OS ^ S S S C^ 00 lo 00 -^ N g ^ g o o S o I o CO -^ CO CO to CM T^ CO -H O O -H g I g IIII M SSS g o I l l l l So ss I i o o S U M I I I! IIII o o i S i g g g g g Mill I I g S I -H M O rh -H CJ g -^ >- '='*'«^ ] CIS ^3 co 00 CD CD S <M en O5 t- --I -H CO t.^ (M Cfl O O O <M S CS O 3 S 3 g t; g I O CO CO CO CO O I l l l l S I = g g 3 S g?5 IIII o» S S S S g l S l l l i l»- 5 S g g I " SSS ggs Sgg g ^ uu i^j ^^ ^'.j *^j C^ CO O O T-H g g g ^ g^! «^ ^ ;=! S S g (M to I l o o Illll '-' S ^ ^ I l l l l - <M CO -* MS CO 3 o * lo CO * o : Si g 56 o o S g g

15 INTERNAL CONTROL JUDGMENTS 57 TABLE 4 Examples of Judgment Consensus among Auditors Relying on Same and Different Questions Most Xmportunt Question 5 5 &6 6 6 Correlation Matrix Auditor No. Auditor No ns ns ns ns (tfi Value for Question No ns *.09.0 ns us ns Auditor No ns" ns Not significant at the.05 level.

16