Applying process mining in a performance audit: A case study

Jans et al. (2013) consider that, in order to convince auditors to adopt process mining as another technique in their toolkit, it is necessary to make a compelling case that process mining is a new way of doing auditing and not just a new way of doing statistical analysis. Given that process mining is costly in terms of effort and skill acquisition, it has to be seen as reducing the workload of auditors elsewhere and/or enabling them to make observations that could not be made otherwise.

In public sector auditing, a great part of the work goes beyond checking the accuracy of financial data, to assess the efficiency of an organization’s processes or the effectiveness of policy implementation (Fluxicon, 2020). Additionally, unlike most business processes, which are relatively standardized, public sector activities are intrinsically diverse and are subject to constant changes. Process mining is therefore of great benefit in the public sector because it can be used to discover processes that auditors are not entirely familiar with—for instance, the processes supporting a newly implemented social program.

The following example illustrates the application of process mining in a performance audit conducted by the European Court of Auditors (ECA; ECA, 2019a). The audit assessed whether the European Union (EU) Commission’s public consultations were effective at reaching out to citizens and stakeholders and making use of their contributions. These public consultations allow European citizens to express their opinions on EU actions and initiatives. The ECA used the audit of this relatively simple consultation process as a “proof of concept” opportunity to apply process mining in auditing.

The consultation process is divided into three phases, and each has several consecutive steps (ECA, 2019a). The phases are: establishing the consultation strategy, conducting consultation work, and informing policy-making (Figure 3).

Since the consultation process consists of a sequence of predetermined steps, each with their respective deadlines, it was possible to analyze the process using process mining techniques. The data received from the European Commission and used as an input in the process mining software consisted of an Excel file with columns containing the consultation ID (case), the steps of the public consultation (event), and start and end dates (time-stamp). Therefore, it was a well-structured small dataset (ECA, 2019b; Fluxicon, 2020).

The process visualization provided by the process discovery technique made it possible to understand the full consultation process and identify anomalies without background information. In Figure 4, which shows a simplified visualization of the process model, the steps in dark blue represent the most frequent activities: roadmap or inception impact assessment (consultation strategy), open public consultation or OPC (conducting consultation), factual summary report (informing), and synopsis report (informing).

The conformance checking technique was also applied, comparing the automatically inferred model against the event log data, indicating how the actual behaviour differs from the process defined by the European Union Commission. This analysis revealed that three public consultations (out of 26) did not have a synopsis report, and that two consultations had additional steps after the legislative proposal (Figure 5). Consequently, it was concluded that these cases deviated from the described processes.

To summarize, in this case study, process mining added value in two ways: (1) it simplified the understanding of the design of the Commission’s framework for public consultations and (2) it helped the audit team to identify deviations from the normal consultation process. This case study presented a very simple case. Auditors should keep in mind that process mining can be applied to processes that are much more complex and to millions of transactions.

Like the European Court of Auditors, other public sector audit offices have started to use process mining techniques in their performance audits. In Canada, the federal Office of the Auditor General of Canada (OAG Canada) has been building its capacity to use these techniques for several years. It has produced a number of performance audit reports that included findings and conclusions based on analysis carried out using process mining. For example, process mining was used in an audit of Immigration and Border Services Canada (OAG Canada, 2017a) to analyze data collected by IT systems at border crossings and identify irregular patterns potentially indicating corruption of border agents. Other examples include two audits of call centres (OAG Canada, 2017b; 2019), in which auditors analyzed event logs for millions of calls placed by Canadians to obtain federal services to determine the efficiency and effectiveness of call centres in responding to these calls.

Page 2 of 3