Audit Tips

Only do a survey if you have a clear rationale

Surveys have become omnipresent in our lives. It seems that every day, whenever we visit a website or buy a product or a service, we are asked to fill out a survey. And, of course, every so often the phone rings and it’s another survey. It becomes tedious… and so we ignore surveys more and more.

There are two lessons in this. First, don’t do a survey in your audit unless you have a clear need and rationale. Second, give careful thought to the population that is going to be filling out the survey and why they would want to provide their opinions to you. Just as you need a reason to conduct a survey, your respondents need a reason to complete it.

Avoid doing a survey early in your audit

A survey is not optimal at the planning or “knowledge of business” stage of an audit. To obtain the most value from a survey, wait until you have completed much of the audit fieldwork and have a good understanding of the auditee’s business practices. This will ensure you can ask questions that are aligned with the audit criteria and are relevant to any performance gaps you discovered during the audit.

This means that you should not use a survey to “fish” for findings at the beginning of an audit project. Instead, you should conduct the survey near the end of fieldwork to confirm the risks or weaknesses you identified and learn how these are perceived by the surveyed population.

Online surveys are not always best

While there are many ways to conduct surveys, we often think of online surveys first. But don’t turn too quickly to an online survey as the best solution. An online survey can certainly be an efficient and relatively inexpensive way to conduct your survey and get results quickly. However, this approach has many limitations. The most obvious one is that only people with Internet access can complete the survey.

Online surveys are best for brief questionnaires on specific issues and for surveying large populations. A mailed questionnaire works better for a longer survey, or one where you want respondents to provide more written commentary.

Be wary of “free” online survey tools. You get what you pay for. Free online surveys are often quite limited, with highly standardized questionnaires that may work well for a common customer satisfaction survey, but are often overly simplistic in an audit context. Hiring a survey firm with expertise in (and access to) more sophisticated web-based applications is often worth the investment.

Questionnaire design is an art, not a science

Designing a questionnaire and creating well-crafted questions is not easy. A good questionnaire should be clear, valid, and interesting. It is important to design the questionnaire based on an analytical framework. There are many ways to structure a survey question. To get it right, you must be very clear on what you are trying to discover and how you will use the results. You should also do the following:

• Scrutinize each question for relevance and clarity.
• Ensure questions are not biased or leading.
• Avoid using double-barreled questions (questions that ask about two topics). Each question must have a single focus, so that you can clearly interpret the results.
• Include an adequate range of response options, because forcing responses into limited categories may create erroneous data. Allow respondents to select “Don’t know” or “Not applicable” or to skip the question.
• Avoid binary Yes/No responses unless there is a follow-up question.

Questionnaires are often limited in space, so do not waste questions asking anything you may not need or may already know the answer to. For example, while demographic questions are common on surveys (such as sex, age, education, and household income), respondents often dislike answering these questions, which feel invasive or as if they may be used to identify them. You may already know some of these demographic characteristics through your audit work. Unless there is a specific need to stratify the data along these demographics, you don’t need to ask for this information.

Consider the social desirability bias

People lie on surveys. Actual behaviour is often different than reported behaviour. This type of response bias is called social desirability bias. We like to look better than we actually are, even on an anonymous survey!

Because respondents won’t always admit to behaviours that make them look bad, be careful not to word the questions judgmentally in any way. For example, if you ask board members, “Do you ever go to board meetings unprepared or not having read the pre-meeting materials?”, few will admit to this and the large majority will say they are always prepared. Instead, you’ll get more reliable results if you ask board members, “Do you feel there are some members who come to the board meeting unprepared or not having read the pre-meeting materials?”.

Think through all possible results

There is nothing more critical in survey research than getting the questions right! In developing your survey questionnaire, think through your expectations for each question. What will each result tell you and how will it add to the audit findings? Knowing how the survey results will be incorporated into the final audit report is a key part of developing the right questions.

What if the survey results are substantially different than what you are expecting? How would that impact your audit report? These questions are also important for the audit team to consider when developing questions. For example, if the audit fieldwork is finding program deficiencies, you may be expecting that program clients will report dissatisfaction. But what if the survey results indicate a high level of satisfaction among clients? What does that mean, or what could be causing that? Would it impact your audit findings, or require you to do more audit work to understand the deficiencies?

All survey results must be reported

All survey results must be reported You cannot pick and choose what survey results to report. It would be highly unethical for auditors to only use the results that agree with what they want to report. All survey results, and often the full survey questionnaire, should be provided in an appendix to the audit report (for an example, see Appendix E in this Auditor General of Manitoba audit report).

Survey respondents will expect to see what the overall results are, and are often curious to see how their perspectives fit in with others’ views on various survey questions. In addition, sending all survey participants a copy of the audit report once published is a good way to thank them for their contribution.