Appendix 1 – Sample Size Determination for Generalizable Sampling

Sample Size

Several factors influence the sample size required for an audit:

• level of detail for reporting
• required level of precision
• expected or observed error rate
• population size

Level of detail for reporting

Level of detail is the degree to which auditors wish to drill down into data and report on segments of the sample as well as overall results. Obviously, a high level of detail requires increasing the total sample size. In situations where a very large overall sample is required, efficiencies can be gained through stratification (i.e., a sample per subpopulation) and planning the minimum sample required for each reported finding. Weighted proportions can be used when calculating overall findings.

Any reported finding needs to have a large enough sample that meets the auditor’s requirement for precision and reliability of results. Auditors should realize that starting with a minimal sample size does not allow for any drilling down of results. If detailed results are required for appropriate observations and recommendations, then sufficiently large samples must be planned to accommodate reporting of subsets of sample data. But if not, a small sample may be sufficient.

Required level of precision

The level of precision needed of any sample depends on the topic of the line of enquiry. If the objective is to determinate, for example, if there is some non-compliance with a specific requirements and the exact level of error is immaterial, then a small sample will be enough.

In situations where smaller levels of error need to be detected or in situations of high importance (e.g., human health and welfare are directly affected), a confidence interval of 5% with a confidence level of 90% would be more appropriate. In the most extreme cases, where the subject is of high materiality and there is a need for a very accurate measure of error rate, then a confidence interval of 5% and confidence level of 95% would be more appropriate. Table 1A provides further information and examples for each of these situations.

Table 1A – Three Levels of Precision for Audit Samples Based on Relative Materiality

*Two sample sizes are provided assuming a large population and two different levels of expected error (EE): 5% expected error and 20% expected error. Population size used for these calculations is 1,000 units. Sample sizes were calculated using CaseWare Analytics IDEA, version 10.

Expected or observed error rate

There are two strategies auditors can use to decide on an estimate of expected error. The first is to assume the highest possible level of variance; that is, an error rate of 50%. This maximizes the sample size. It is an easy and safe strategy, but also the costliest.

An alternative strategy is to consider how different levels of error affect conclusions and recommendations. Starting at 0% error, auditors postulate conclusions and recommendations for increasing levels of error. At some point, the increasing level of error no longer has any impact on conclusions or recommendations. Depending on the situation, that point might be an error rate of 10%, 20%, or 30%.

To illustrate the effect of expected error, several scenarios are depicted in Table 1B. Three options for expected error are presented: 5%, 20%, and 35%. As the expected error increases toward 50%, the respective sample sizes also increase—and so does the amount of variance. In order to maintain the same level of precision (5% confidence interval), the sample size must increase to compensate. For these examples, we have chosen a 5% confidence interval at a 90% confidence level as the required level of precision.

Table 1B – Three Levels of Precision for Audit Samples Based on Relative Materiality

*Sample sizes were based on the following parameters: Population size of 1,000, confidence level of 90%, and confidence interval of 5%. Sample sizes were calculated using CaseWare Analytics IDEA, version 10.

Two potential observed error rates are presented for each case; one below the expected error, and one higher than the expected error. When the observed error is below the expected error, the observed confidence interval is smaller, more precise, than the required level. As a result, the observed error rate can be reported in an unqualified manner.

When the observed error rate is higher than the expected one, the observed confidence interval is now larger, less precise, than the required confidence interval. As a result, the sample is no longer large enough for the specified confidence interval and confidence level; it is not reportable at the accepted confidence level and confidence interval. A possible solution may be to increase the sample size, time and resources permitting. It is important to keep in mind that a higher error rate indicates other potential audit issues, which may warrant reassessing the overall risk assessment.

A lower expected error does result in a smaller sample size, and is therefore less costly, but it also limits the range of possible audit observations. The strategy of always choosing a 50% error rate provides the greatest flexibility in reporting but it also maximizes the sample size and increases the audit’s cost. A more reasonable strategy is to choose the lowest possible error rate that would trigger the most drastic recommendation; that is, the point at which an error becomes so large that the most drastic recommendation is warranted.

Population size

Sample size is only mildly influenced by population size, and usually only when the population is very small. This is because variance in a population quickly reaches a plateau as the size of the population increases. That is, the amount of variance in a population of 1,000 is about the same as the variance in a population of 3 million. As a result, both populations require similar sample sizes despite the differences in population size. For extremely small populations, the required samples sizes diminish slightly. Regardless of population size, sample size should always be calculated in the same manner. A good audit data analysis software will have a built-in utility for sample size calculation.

• PreviousReporting the Results from a Purposeful Sample
• NextCalculating Sample Size