Step 1: Determining why  you need a survey

Before conducting a survey within a performance audit, you need to be very clear on why you are doing it. What is the link to your audit objective? What are you seeking to discover? Is conducting a survey the only way to get this information?

Usually, a survey is only one information-gathering technique used in a performance audit. Like other techniques, a survey should support the assessment of the audit criteria, and provide a basis upon which to draw audit conclusions. Therefore, surveys should be conducted only after the audit criteria have been established and not be used as an early planning tool to help your audit team decide what to focus the audit on. For example, you should not conduct a survey at the beginning of an audit to determine what a program’s risks are. Instead, you should conduct the survey later on to confirm the risks identified by the audit and to find out how these risks are perceived by the population being surveyed.

Step 2: Determining what  information the survey will provide

Be clear on what information you will be able to provide from the survey results. What are you trying to measure? How will the results be used within the audit findings? Having a good understanding of this will help you decide whether to do the survey and how best to design the questions that need to be asked.

You should have realistic expectations of what insights the survey results can provide. Surveys have limitations and it is important to understand that survey results are not audit evidence. Surveys are opinion-based; they provide a snapshot of respondents’ perceptions about something at a point in time. They do not provide objective truth, but rather information on how respondents perceive reality. For example, if all members of a board of directors rate their board as highly effective in a survey, it does not mean that the board is actually an effective one; it simply means that they perceive it to be so. Auditors should always be mindful of this nuance when reporting on survey results.

Step 3: Determining how  the survey will be conducted

Once the audit team is clear on the why and the what, then decisions can be made on how to effectively conduct the survey. What works best will depend on the answers you came up with in Step 1 and Step 2, and on what you are measuring. You need to know the limitations of each approach (online, mail, individual interviews, focus groups, etc.) in order to determine what would work best for what the audit is trying to assess or measure. When people hear the word “survey”, they most often think of a structured questionnaire, which can be a paper-based mailed survey or an online survey. These types of questionnaires are commonly used for large populations and provide mostly quantitative results, although there should always be room for qualitative answers as well.

Internet surveys are ubiquitous in today’s world. In fact, they are overused, and respondents are often wary of them, questioning whether their confidentiality is truly being protected. Survey fatigue also sets in very quickly for online surveys, and respondents are more likely to abandon them before finishing, which limits the number of questions you can ask and the amount of information you can gather. For these reasons, online surveys do not always lend themselves well to a performance audit.

In some cases, the most effective approach may be to use a fully qualitative process that is conducted through structured interviews. This is a good option if the population being surveyed is small, because conducting qualitative interviews takes a significant amount of time and resources. A focus group is another common qualitative approach. Most often, this is used for conducting product or market research, or testing reactions. But a focus group can also be used alongside a structured questionnaire to follow up and clarify the meaning of quantitative survey results. Another option is to deliver a group questionnaire through a moderator, with the group discussing their responses, or, if quantitative results are needed, the group members can provide their responses through electronic voting.

Step 4: Designing the survey questionnaire

Once you’ve chosen a survey approach, the next step is to design the survey questionnaire—the hardest part of the process and the most critical to get right! Survey questionnaires are best developed using an analytical framework, based on the audit criteria, that fully outlines what is being measured and assessed. Populating all the possible questions within this framework will allow you to eventually narrow down and select the questions that work best for your audit. This method also ensures that the survey results will align with the audit objective and criteria, and that the information you collect through the survey will be useful in preparing the audit report.

It is best to have completed as much of the fieldwork as possible on other aspects of the audit before finalizing the survey questionnaire. This allows you to consider any program gaps or weaknesses identified when developing relevant questions where the respondents’ perspectives can add value.

Survey questionnaires ideally include a variety of question types, both open-ended and closed, to keep respondents engaged. There are many different types of close-ended survey questions, including asking respondents to:

• assess their level of agreement or disagreement with a statement (using a Likert scale),
• rank their preferences, or
• select from a multiple-choice list of possible answers.

In contrast, open-ended questions allow respondents to provide answers in their own words. These are important to have but must be used carefully because they can add to the costs of the survey and may not provide useful information if not asked well.

Questionnaire design is an art and it is much harder than it seems. Errors in design can have regrettable consequences; for example, faulty questionnaires can generate misleading results. For this reason, it is extremely important to think through how the survey results are going to be reported, so that care can be taken in wording the questions accordingly. Survey questions must be clear, with a single focus, and not be leading or biased. There is also an important psychological component to question sequencing. Questions at the start of a survey must be engaging to draw the respondents in and make them want to complete the survey. Don’t ask overly difficult or intrusive questions at the beginning of the survey, as respondents may feel discomfort and abandon the questionnaire.

Finally, you should always pre-test the survey on a small subset of respondents or people who present the same characteristics as the survey population to ensure they clearly understand the questions. Then you can clarify any questions and work out any other difficulties, such as sequencing issues, before sending the survey to the full population.

Step 5: Ensuring the survey results are reliable

Even the best-designed survey will not prove very useful if it is not sent to the appropriate respondents or if only a few respondents complete it. For survey results to be reliable, the target population must be chosen carefully and the response rate must be high.

Sampling a population effectively and ensuring you receive results you can rely on is always a challenge when conducting surveys. Whenever one tries to generalize the results of a survey that was conducted on a sample population (only a part of the overall population), there is always a margin of error. A calculation of this margin of error is an important part of understanding how much the survey results can be relied upon. You should keep this margin of error as low as possible by using appropriate sample selection techniques.

In a performance audit, it is recommended to send the survey to the entire respondent population whenever possible. This is called a census approach, and it is always preferable because it provides the most reliable results. You should use a sample only when the entire population is not known or not accessible, or when a census approach would be too resource-intensive.

Achieving a high survey response rate is also important to ensuring you get valid results. In order to ensure a good response rate, it is critically important to assure respondents that their confidentiality and anonymity will be fully protected. Legislative audit offices are in a highly privileged position to conduct surveys. Our credibility, independence, and political neutrality provide us with a certain amount of authority, moral suasion, and trust, and we must ensure that that trust is never broken.

Why should you hire experts?

Surveys may seem easy, but they often are not. Because auditors are not usually experts in survey methodology, consider hiring a survey firm to assist you, especially if the audit or the information being sought is sensitive in any way.

Before going out to procure a survey firm, you should keep two things in mind:

1. For a legislative audit office’s work, political neutrality is crucial. Many survey firms are tied to specific political parties or do political polling, so make sure you are using a firm that does not work for political parties or have any political affiliation.
2. Cheapest is not always best. Be careful of the expertise you are buying. If the “experts” are simply using the same online tools you could have used, then you could likely have done it yourself.

Once you select your survey firm, rely on their expertise. Experts can help you select the best methodology for your purpose and should be able to tell you if you are going in the wrong direction for what you wish to achieve. They can also assist you in questionnaire development and design. They will advise you when a question is too leading, or when you are influencing or biasing the results. They will help you conduct the survey pre-test and make corrections where required. They will also help you achieve a good survey response rate. In the end, their expertise should provide you with reliable results that contribute to a sound final report.

However, you should stay involved in the research throughout. Do not just hire experts, hand off the project to them, and hope for the best. Give them clear objectives and share your audit criteria with them. Explain exactly how you will want to use the survey results in your audit report. With this information in hand, they will help you interpret the survey results with an appropriate lens and make sure you don’t overreact to the results or overstate them.

Conclusion

A survey is not an audit, but it can be a very useful element in a performance audit. Opinion surveys provide important perspectives that can help supplement audit findings. When done correctly, using appropriate methodology, rigour, and expertise, surveys can provide reliable information that adds significant value to a performance audit report.

References

Auditors interested in reading an audit report that used survey methods designed around the audit’s criteria can consult the 2014 Auditor General of Manitoba’s audit report on Manitoba’s Framework for an Ethical Environment. Appendix E in this report presents the survey methodology that was used and the detailed survey results.