Is Your Sample Representative? Are You Sure? May 8, 2014 Beth A. Wood, CPA, North Carolina State Auditor 1
Is Your Sample Representative? Course Objectives Understand Some Fundamental Concepts of Sampling Understand What is Meant by Representative Sample Understand How to Ensure Samples Are Representative Understand Obstacles of Dual-Purpose Testing 2
What Audit Sampling Is Method for Obtaining Audit Evidence Based on Statistical Theory Used to reach a conclusion, about a large population by examining only a portion of the data 3
Where Do We Use Sampling? Tests of Controls Substantive Tests Compliance Tests Dual Purpose Tests 4
Types of Audit Tests Tests of Controls provides evidence about the design, implementation or operation of a control Preventing or Detecting Material Misstatements or Noncompliance Test of Controls- Financial Statement Audit or Compliance Audit Auditor is generally concerned with rate of deviation from prescribed control procedures. 5
Types of Audit Tests Substantive Tests provides evidence about validity & propriety of the accounting treatment of transactions or balances or to detect misstatements. Auditor is interested primarily in a conclusion about dollars. 6
Types of Audit Tests ComplianceTests provides evidence about whether the auditee has complied with laws, regulations, contracts and grant agreements pertaining to federal awards that may have a direct and material effect on each major program identified for audit. Auditor is concerned about whether or not there is evidence of compliance (rate AND likely magnitude of noncompliance). 7
Types of Audit Tests Dual-PurposeTest Financial Statement Accounts/Class of Transactions - a test designed to test the effectiveness of a control and test whether a recorded balance or class of transactions is materially misstated using the same sample. Compliance a test designed to test the operating effectiveness of a control and test whether the auditee complied with relevant laws, regulations, or provisions of contract or grant agreements using the same sample. 8
Sampling Terminology/Concepts (Statistical) Precision- how close an estimator is to the true value of a parameter (in this case parameter is the true value/rate within the population) Sampling Risk The risk that the auditor s conclusion, based on a sample, may be different from the conclusion if the test were applied the same way to the entire population. Reliability/Confidence Level of Assurance Complement of Sampling Risk, i.e., if Sampling Risk is 5% then Confidence Level is 95%; If Sampling Risk = 10% then Confidence Level is 90% 9
Sampling Terminology/Concepts Statistical vs Nonstatistical Statisitcal Sampling Uses the law of probability to explicitly measure sampling risk associated with a sampling procedure. Confidence/Reliability AND Precision MUST use appropriate random sampling method Simple Random Sampling Systematic Sampling 10
Sampling Terminology/Concepts Statistical vs Nonstatistical NonStatisitcal Sampling Does not measure sampling risk Can be as effective as statistical sampling Sample Size would ordinarily be comparable to effectively designed statistical sample. Warning Professional Judgment May Deviate from Nonstatistical Sampling Guidance and Known Statistical Sampling Concepts 11
Is Your Sample Representative? 12
Define the Population Tests of Controls Population is all transactions to which the control should be applied in a given time period Need to define the period for which evidence about the control is needed Cannot project to a population that the sample was not extracted from 13
Define the Population Most common populations are: All transactions in the fiscal year (used when relying on controls for balance sheet and/or income statement accounts) AND for program transactions; All transactions in an account balance at year end (used when relying on controls for balance sheet accounts only); 14
Define the Population (cont.) Test of Controls - Considerations Need to ensure the population is complete Reconcile population from which sampled selected Population of Controls over Major Programs 15
Define the Population (cont.) Substantive Tests- Considerations Treatment when considering/testing Individually Significant Items - FS Reconcile population from which sampled selected 16
Define the Population (cont.) Substantive Tests- Considerations Treatment when considering/testing Individually Important Items Compliance Not possible to test compliance across major programs Difficult to test multiple compliance requirements with one sample 17
What if This is Your Result? Controls/Compliance Selection Attribute 1 1 Yes 2 Yes 3 N/A 4 N/A 5 N/A 6 Yes 7 Yes 8 Yes 9 Yes 10 Yes 11 Yes 12 Yes 13 Yes 14 Yes 15 Yes 16 Yes 17 Yes 18 Yes 19 Yes 20 Yes 21 Yes 22 Yes 23 Yes 24 Yes 25 Yes 22/25 WHAT IF REASON WAS: 1. Voided Checks Verify, document, replace selections to obtain a full 25 items to test 2. Missing Documentation Document, consider underlying reason Control not occurring as intendedfinding, or Existence of another population-more testing 3. Population Not Properly Defined DO NOT just continue to pick items without understanding the cause. 18
Is Your Sample Representative? 19
Is Your Sample Representative? 20
Choose Audit Sampling Technique Statistical VS Nonstatistical Does It Make A Difference? Yes & No 21
Determine the Sample Size Test of Controls-Financial Risk of Overreliance (Sampling Risk) Tolerable Rate of Deviation Expected Rate of Deviation Tricky Subject to Manipulation 22
Determine the Sample Size Test of Controls-Financial Sampling Risk the risk that the auditor s conclusion based on a sample might be different from the conclusion if all items in population were sampled Risk of Assessing Control Risk too High Risk of Assessing Control Risk too Low Complement to Desired Level of Assurance (Confidence) 23
Determine the Sample Size Test of Controls-Financial Risk of Assessing Control Risk too High Relates to the Efficiency of the Audit May lead auditor to increase scope of substantive tests unnecessarily Risk of Assessing Control Risk too Low Relates to the Effectiveness of the Audit May lead auditor to inappropriately reduce the evidence obtained from substantive tests Potentially more serious 24
Determine the Sample Size Test of Controls-Financial Tolerable rate = the maximum rate of deviation that will still support the control risk assessment Matter of audit judgment Tolerable rates contemplate that not every deviation results in a misstatement or noncompliance with program requirement The Lower the Tolerable Rate the Larger the Sample Size Greater Degree of Reliance on Controls by Auditor the Lower the Tolerable Rate 25
Determine the Sample Size Test of Controls-Financial Tolerable rate: Planned Assessed Level Tolerable Rate Of Control Risk % Low 3-7 Moderate 6-12 Maximum Omit Test 26
Determine the Sample Size Test of Controls-Financial Expected Deviation Rate - Auditor Considers: Results of Prior Year s Tests, The Design of Internal Controls The Control Environment Note: the auditor considers prior year s results in light of changes in entity s internal control & changes in personnel. For a given Tolerable Rate, Sample Size Increases as Expected Population Deviation Rate Increases 27
Determine Sample Size Substantive Tests of Details-Financial Determine Degree of Reliance Determine Tolerable Misstatement Estimate Expected Misstatement Determine population balance after testing ISI Adjust Sample Size for Stratification (for nonstatistical) 28
Determine Sample Size Substantive Tests of Details-Financial Determine Degree of Reliance: Assess the risk that other substantive procedures designed to test the same assertion will fail to detect material misstatement Maximum no other tests performed Moderate other tests moderately effective in detecting material misstatements Low other tests highly effective in detecting material misstatements 29
Determine Sample Size Substantive Tests of Details-Financial Determine Degree of Reliance: (cont d) Consider the Level of Inherent Risk for the particular assertion(s). Consider the effectiveness of the controls in preventing and detecting material misstatements. Combine the 2 using the following categories: Maximum Slightly Below Maximum Moderate Low 30
Determine Sample Size Substantive Tests of Details-Financial Determine Tolerable Misstatement: The Maximum Error in the Population the Auditor is willing to accept; Generally less than Materiality as a Whole for the Engagement; Determined in Planning Sample Sizes Increase as Tolerable Misstatement Decreases 31
Determine Sample Size Substantive Tests of Details - Financial Determine Expected Amount of Misstatement: Total Amount of Misstatement Expected to Find in the Population; As Expected Amount of Misstatement Approaches Tolerable Amount There is a Need for More Precise Information From Sample; Therefore, Resulting in Larger Sample Size Result of Planning 32
Determine Sample Size Test of Controls - Compliance Required to Plan to Obtain High Assurance (As If Going to Rely On) Sample Table Inputs Used by AICPA GAS/A-133 Guide: 90-95% Confidence 5-10% tolerance for deviations Plan for Zero deviations Significance of control/inherent risk of noncompliance 33
Determine Sample Size-Test of Controls - Compliance Suggested Minimum Sample Sizes Significance of Control and Inherent Risk (IR) of Compliance Requirement Minimum Sample Size 0 deviations expected Very Significant and Higher IR 60 Very Significant and Limited IR Or 40 Moderately Significant and Higher IR IR = Inherent Risk Suggested minimum sample sizes for populations >250 34
Determine Sample Size Compliance Suggested Minimum Sample Sizes Desired Level of Assurance (Remaining Risk of Material Noncompliance) Minimum Sample Size 0 deviations expected High 60 Moderate 40 Low 25 Suggested minimum sample sizes for populations >250 35
Is Your Sample Representative? 36
Sample Selection Controls/Substantive/Compliance Random-Number Sampling Systematic Sampling To be used with Statistical Sampling must have a random start Block Sampling (Nonstat only) Haphazard (Nonstat only) 37
Substantive Tests of Details 7. Evaluate: Project Results Consider Sampling Risk- Example: TM for the balance of AR is $30,000 Total AR balance is $700,000 Projected error is $7,000 (point estimate) Is there an acceptable level of sampling risk? What if the projected error was $28,000? 38
Is Your Sample Representative? 39
What Mishaps That Prevents Representative Sample 40
Is Your Sample Representative? 41
Define and Select: Dual-Purpose Sample Considerations Common practice to utilize a single sample to achieve multiple audit objectives Internal control over compliance testing Compliance testing Financial statement balance testing Exercise caution: Different characteristics are for different objectives Need to use the larger of the sample sizes determined for the separate tests If there are errors in internal control, compliance sample may not be adequate 42
Dual-Purpose Sampling: Where Does this Go Wrong? I m selecting a sample to test a control. The control operates over all entity expense transactions. I want to be sure to select some transactions from each of the major programs. If I select a sample from the more limited population of just transactions charged to federal programs, can I extrapolate the results to all entity expense transactions? Alternatively, can I allocate a portion of my sample to federal program transactions first, and then supplement with transactions from the rest of the population? 43
Dual-Purpose Sampling: Where Does this Go Wrong? Can I triple- test (FS, IC and Compliance) major program transactions? Maybe. It depends. There are 4 major programs that all use the same review and approval system. Can I dual-test controls and compliance across all four programs? Controls- Yes. Compliance- No. The major program CFDA # comprises two very different programs. One program is 87% of the total expenses; the other program is very small. Can I only test the larger program? No. 44
Is Your Sample Representative? Questions? 45
Sampling in Tests of Controls 1. Determine Test Objectives 2. Define the Deviation Condition 3. Define the Population 4. Define the Sampling Unit 5. Choose Audit Sampling Technique 6. Determine the Sample Size 7. Sample Selection 8. Perform Audit Procedures 9. Evaluate Sample Results 46