The importance of corroborating survey results

One problem with employee surveys is that the internal group that owns them often has no resources to perform any follow-up investigation; the survey results, after a superficial analysis, are taken to be definitive. Yet, notwithstanding that employee perceptions can be quite mistaken, we also know that survey results are subject to shortcomings and biases. For example, employees may be less than candid if they believe they can be identified, or they may be more motivated to answer the survey if they are dissatisfied, or less motivated if they believe there will be no follow-up of results. In addition, analytics can reveal interesting associations, but association is not causation.

Although the survey results analyzed through the Python-based approach suggested in this article provide illuminating insights to auditors, they do not provide them with stand-alone evidence to reach audit conclusions. Auditors must also look for corroborating evidence. They must complement surveys with their own observations from interviews, documents, and other sources.

Survey design considerations

This Python-based analytical approach can be applied to almost any employee survey to bring out important information that otherwise would remain hidden in the data. However, this approach has two dependencies that should be kept in mind in order to enhance its applicability. As necessary, auditors should be prepared to work with management to take these dependencies into account during the design of employee surveys.

1. Although the Gini function does not strictly depend on the number of groups, it will produce more reliable results when there are more employee groups. Twenty groups seems to be a good number because that results in distributions in a two-by-two matrix that are readily discerned by the eye. Hence, demographic questions should be set up to distinguish around 20 or more mutually exclusive groups.
2. This analytical approach depends on the possibility of finding meaningful associations between the responses to survey questions. To this end, the more comprehensive the survey is of the various aspects of organizational culture, the more effective the approach will be. The questions should be based on a good model of organizational dynamics and organized into themes. A good example of a survey designed in this way is the Public Service Employee Survey undertaken annually by the Government of Canada.

It may not be practicable to cover all possible themes or all organizational groups in one survey. In this case, provided that either the organizational groups match or the questions match and the groups are all mutually exclusive, surveys done separately can be combined later. This can usually be done fairly easily using a spreadsheet program, but if the process is to be repeated, it would make sense to do it programmatically using Python.

It is recommended that surveys use a five-point response scale that ranges from “strongly disagree” to “strongly agree”. However, sometimes separate surveys might use differing response scales; for example, a four-point scale instead of a five-point scale. In this case, provided that the organizational groups match, the surveys can nonetheless be combined after converting the four-point aggregate scores to their five-point equivalents. This conversion is a trivial mathematical operation that can easily be handled by Python.

Where to from here?

It is clear that Python can be used to code innovative modules that apply analytics to employee survey data for use in culture audits. Results obtained with this technique can lead to new insights or corroborate findings from more conventional audit techniques. They can also support root cause analysis and facilitate the formulation of actionable, well-targeted recommendations, which is the goal of internal auditing.

What is still to be developed is a survey analysis package that would then be published for sharing and review by the Python open-source community. This publication project can only be accomplished through a future collaboration between internal auditors, performance auditors and Python coders. As an experienced auditor and intermediate Python developer, I am actively striving to bring about this collaboration. It is my hope that this article will contribute to that endeavour.

Conclusion

Because culture is an organization’s most important soft control, auditing organizational culture deserves to be taken seriously. Regular employee surveys can make an important contribution to culture audits, but some of their value remains hidden unless data analytics is used to find the associations between employee responses. Data analytics can be conveniently implemented using Python; however, this will require collaboration between internal auditors, performance auditors and Python developers. This article aims to contribute to an advancement of this collaboration.

References

Anyaeche, A. (2020, June 2). Programming for Accountants. The Start-up. Available at https://medium.com/swlh/programming-for-accountants-27f4d48d722d

Bhuyan, D. (2019, January 10). Auditing the Data Using Python. LinkedIn. Available at https://www.linkedin.com/pulse/auditing-data-using-python-dhiraj-bhuyan-msc-b-tech/

Canadian Audit and Accountability Foundation (2019). Artificial Intelligence and Auditing: Overview of Potential Impact on Public Sector Auditors. Available at https://www.caaf-fcar.ca/en/performance-audit/research-and-methodology/research-highlights/3455-research-highlights-3

Canadian Audit and Accountability Foundation (2020a). Auditing Organizational Culture in the Public Sector. Available at https://www.caaf-fcar.ca/images/content/performance-audit/ResearchHighlights/PDF/AuditingOrganizationalCulture-PublicSectorEN.pdf

Canadian Audit and Accountability Foundation (2020b). Better Integrating Root Cause Analysis into Public Sector Performance Auditing, second edition. Available at https://www.caaf-fcar.ca/images/pdfs/research-publications/RootCauseAnalysisEN.pdf

Capozzi, M. (2020). Asking the Right Questions: Conducting Surveys in Performance Audits. Canadian Audit and Accountability Foundation. Available at https://www.caaf-fcar.ca/en/performance-audit/research-and-methodology/voices-from-the-field/voices-issue-8

CPA Canada. (n.d.). Why Should CPAs Code? Available at https://www.cpacanada.ca/en/business-and-accounting-resources/other-general-business-topics/information-management-and-technology/publications/coding-important-for-cpas.

Dogas, C. (2020). Internal Audit Still Lags in Use of Data Analytics. Internal Audit 360°. Available at https://internalaudit360.com/internal-audit-still-lags-in-use-of-data-analytics/

Government of Canada (2019). 2019 Public Service Employee Survey Results by Theme for the Public Service. Available at https://www.tbs-sct.gc.ca/pses-saff/2019/results-resultats/bd-pm/00/org-eng.aspx

IDEA. (n.d.). 3 Must-Have Features of an Audit Data Analytics Solution in 2020. Available at https://idea.caseware.com/blog/audit-data-analytics-3-must-have-features

Institute of Internal Auditors (IIA) (2016). Auditing Culture: A Hard Look at the Soft Stuff. Global Perspectives and Insights, Issue 3. Available at https://global.theiia.org/knowledge/Public%20Documents/2016-Feb-GPI-English.pdf

Matplotlib (n. d.). Creating Annotated Heatmaps. Available at https://matplotlib.org/3.1.1/gallery/images_contours_and_fields/image_annotated_heatmap.html

Pett, J. & S. Pundmann. (June 2016). Care and Feeding of the Company’s Culture. Available at https://www2.deloitte.com/us/en/pages/audit/articles/care-and-feeding-of-the-company-culture.html

Python Software Foundation (2021). About Python. Available at https://www.python.org/about/

Roth, J. (March 2011). Soft and Strong: A Best-Practice Paradox. Tone at the Top, Issue 50. The Institute of Internal Auditors.

Roth, J. (March 2019). Auditing Culture: History and Principles. IIA Journal, The Institute of Internal Auditors.

Roth, J. (May 2019). Auditing Culture: Bumps in the Road. IIA Journal, The Institute of Internal Auditors.

Roth, J. (June 2019). Auditing Culture: Where to Begin. IIA Journal, The Institute of Internal Auditors.

Roth, J. (August 2019). Auditing Culture: Observation and Data. IIA Journal, The Institute of Internal Auditors.

Roth, J. (November 2019). Auditing Culture: Audit Project Surveys. IIA Journal, The Institute of Internal Auditors.

Roth, J. (January 2020). Auditing Culture: Employee Surveys. IIA Journal, The Institute of Internal Auditors.

Roth, J. (December 2020). Auditing Culture: Conducting an Entitywide Survey. IIA Journal, The Institute of Internal Auditors.

Seago, J. (June 2016). Toxic Leaders, Toxic Culture. Internal Auditor.

Tabuena, J. (2016, 23 July). Auditing Corporate Culture: A New Imperative. MIS Training Institute. Available at https://www.experis.ca/Website-File-Pile/Whitepapers/Experis/auditing-corporate-culture.pdf

Tahsildar, S. (2019, 18 April). Gini Index for Decision Trees. QuantInsti’s blog. Available at https://blog.quantinsti.com/gini-index/

Page 3 of 3