Analytical Evidence

Many different procedures can be used to generate analytical evidence in support of audit observations on oversight. Some can be relatively simple, like reviewing the minutes of board meetings over a precise period of time in order to determine whether meetings are held regularly, whether directors have a good track record of attending the meetings and what topics were discussed by the board. Other procedures are more complex and will often require the assistance of specialists, like surveys and benchmarking exercises. This section provides information on surveys and benchmarking.

Surveys

Conducting surveys is a useful audit procedure when the scope of an audit of oversight is large, covering multiple organizations or a whole sector (health or education, for example). Surveys enable auditors to collect specific, structured information from a well-defined population.

In audits of oversight, surveys can be used to obtain information on the policies, systems, and practices in place in different organizations. They can also be used to obtain opinions on oversight body dynamics or on the effectiveness of specific oversight practices and functions.

While surveys can be useful, auditors should note that they are qualitative assessments (especially surveys of opinions) and are not generally sufficient on their own as audit evidence. Indeed, in its 2013 document Crown Agency Governance – Obtaining Audit Evidence: Challenges, the CCOLA Governance Study Group considered that opinion surveys do not generally provide audit-level assurance on a board’s performance, nor on the quality of its oversight. However, it is possible to use data collected through a survey in combination with other types of evidence to provide audit-level assurance on specific audit observations. Also, survey data can be used as the basis of a non-assurance report (see, for example, British Columbia’s 2009 OAG report on information use by boards of public sector organizations and Manitoba’s 2009 OAG study on board governance in Crown organizations).

Finally, auditors need to be aware that surveys are complex procedures that require much thought, time, resources, and expertise. Developing and conducting surveys requires specialized knowledge and skills. For this reason, auditors are encouraged to consult with an internal specialist or an external expert before proceeding with a survey as part of the audit of oversight.

Benchmarking

Benchmarking is a method for comparing performance, systems, or processes across and between organizations, across or between countries. In audits of oversight, benchmarking can be used for three purposes:

1. to identify best practices that will be used as audit criteria,
2. to assess the design of oversight structures and systems and/or the results and effectiveness of oversight bodies compared with those of other organizations or with recognized best practices, and
3. to identify best practices that will constitute the foundation for audit recommendations.

The principal advantage of benchmarking is that it provides an objective basis from which to derive audit observations and conclusions. When properly conducted, benchmarking can allow auditors to reach conclusions on the structures and systems of an organization’s oversight bodies and on their relative effectiveness compared with best practices or with similar organizations in a sector of activity.

However, conducting and documenting the results of a benchmarking exercise can be time consuming and challenging, especially when authoritative sources of best practices are not readily available.

Benchmarking the design of oversight structures and systems will generally be easier to do than benchmarking their effectiveness. Information on mandates, governance structures, policies, and practices can readily be obtained by conducting a survey of similar organizations and by collecting documentary evidence available in the public domain or upon request. Public organizations will often be willing to provide information on the design of oversight structures and systems. Obtaining reliable information on the results and effectiveness of oversight bodies and their practices will usually be more difficult, especially when there are significant deficiencies that selected organizations would rather not bring to the attention of auditors.

Beyond obtaining sufficient information, auditors who want to use benchmarking as a source of evidence must ensure that they are making valid comparisons. They should do the following:

• Compare organizations that operate in the same sector of activity and that share significant operational characteristics. In general, comparing public sector organizations with private sector ones will not be appropriate because of the very different goals pursued by each type of organization.
• Use equally reliable data for all the organizations covered by the analysis. Using only annual reports and website information is insufficient to compare effectiveness unless the reliability of this information is assessed by the auditors.
• Close all information gaps and clear all uncertainties by obtaining documentary or testimonial evidence from selected organizations.

Finally, before embarking on a benchmarking analysis, auditors are advised to consult with the audited organization’s management and with subject experts to discuss which organizations (or countries) would constitute acceptable comparators. It is preferable to obtain management’s agreement with the methodology used, but auditors can expect to run into arguments that the audited organization has unique challenges and cannot fairly be compared with its peers. In such situations, audit teams will need to exercise their professional judgment and decide whether or not to proceed with a benchmarking analysis.

• PreviousPhysical Evidence
• NextReporting the Results of an Audit of Oversight