Area Summary: Procurement of Goods and Services – Public Private Partnerships

Relevant Audits

Examples of Audit Objectives

• To determine whether the Government had adequately managed the risk of entering into a public-private partnership to build the bridge. (View report summary)
• To determine whether the Department had put in place an appropriate framework to manage the key risks associated with the quality, schedule, scope, and cost of the bridge project. (View report summary)
• To assess the business case on which the Department’s decision to adopt the P3 approach for the two school project was based. (View report summary)
• To determine if agencies adequately demonstrated that the recommended P3 arrangement represents the best procurement solution taking full account of the expected costs, benefits and risks across the project’s life-cycle?. (View report summary)
• To determine if the project demonstrates that the P3 approach provides value for money by structuring the project such that:
  • lifecycle costs were minimized;
  • risks were transferred to, or retained by, the party who could most  cost-effectively manage the risk; and
  • processes developed, challenged and  validated significant assumptions  contained in supporting analyses such  as the public sector comparator and the shadow bid. (View report summary)
• To determine if a fair, open and transparent process was used during the procurement. (View report summary)
• To determine whether the Department’s contract management processes and procedures are adequate to ensure services detailed in the service contracts are received and payments for services are made in accordance with the contracts. (View report summary)
• To determine whether service contract terms are adequate to ensure the public interest is being protected. (View report summary)

Examples of Audit Criteria

• Investment objectives and intended outcomes have been clearly defined and aligned with government’s policy objectives. (View report summary)
• The costs, benefits and risks of the project options that could feasibly deliver on investment objectives have been rigorously assessed, verified and clearly communicated to decision makers. (View report summary)
• Project recommendations have been based on the analysis of costs, benefits and risks while taking account of stakeholder consultations. (View report summary)
• An analysis shows that the P3 approach offers the best value for the Department, who ensures that this value is maintained throughout the project, namely during amendments. (View report summary)
• A rigorous process ensures a competitive and fair treatment of proposals throughout the different stages (call for interest, call for qualifications, call for proposals, etc.). (View report summary)
• The Department ensures the financial viability of the project and the proposals received. (View report summary)
• Responsibility and risk sharing between public and private sectors, compensation mechanisms and guarantees allow for the best VFM throughout the duration of the agreement. (View report summary)
• All significant assumptions made by the Department are supported and documented. (View report summary)
• Due diligence is performed to review the value-for-money report. (View report summary)
• The Government manages the risks of entering into a public-private partnership. (View report summary)
• The agreement specifies the type of information the partner must provide in order for the Department to effectively monitor compliance of the project implementation against requirements, as well as frequency of reports. (View report summary)
• The agreement clearly states the process to be followed in the event of a dispute. (View report summary)
• The Department has clearly stated the responsibilities and tasks with regard to contract management. (View report summary)
• Performance monitoring is systematic and based on sufficient and reliable management information. (View report summary)

Examples of Evidence Gathering and Analysis Techniques

File Review

• Reviewing documents developed or used by the Government to support key decisions made on the project between 2000 and 2010. (View report summary)

Analysis

• Comparing management practices against several good practice guides. (View report summary)
• Examining the financial models used to calculate value for money. (View report summary)
• Meeting with consultants specializing on P3s and hired financial modelization specialists to analyse the project’s financial model. (View report summary)
• Testing compliance with service contract terms. (View report summary)

Examples of Findings

• The project was not a public-private partnership. The Concession Agreement assigned most of the project risks to the Government and did not shift any significant risk to the private sector; risk sharing was anticipated when a P3 procurement strategy was selected. (View report summary)
• The Government authorized bridge construction to begin without having the assurance of a fully developed design. As a result, the risk to the project was significantly increased. Ultimately, the inability to resolve design issues within the specified time frame resulted in the lenders declaring the Corporation to be in default and requesting the Government to assume the project debt. (View report summary)
• The Department has a framework in place to manage the key risks, but there are weaknesses in the risk matrix it developed for the project—for example, some of the risk mitigation measures are too general to be useful. Significant risks remain in the areas of the project’s schedule, scope, and cost. (View report summary)
• There was no evidence that a formal preliminary analysis was performed to support the decision of adopting a P3 approach before it was publicly announced. The audit team was unable to determine the rationale for the decision. (View report summary)
• The Department concluded the P3 approach provided better VFM to taxpayers for this project over the traditional approach. However, after adjusting for the effect of the maintenance and lifecycle cost assumption, the traditional model would deliver $1.7 million VFM over the P3 approach. (View report summary)
• There was little documentation available showing that the Department had reviewed the assumptions upon which the VFM analysis was based. (View report summary)
• The audit team noted three areas where the VFM analysis was deficient in comparison with common industry practice. These included:
  • sensitivity analysis (i.e. only risk quantification was subject to this analysis);
  • timing of preliminary VFM analysis (i.e. the analysis was not completed prior to announcing the P3 project); and
  • reporting of VFM results (i.e. the Department did not comply with the government’s P3 protocols that require fair and transparent reporting). (View report summary)
• Increased costs were incurred for a number of reasons. The entity did not have a clear understanding of the scope and user requirements of the project, which resulted in numerous variations in the project and prevented the effective transfer of design and scope change risk to Access Health Vancouver. (View report summary)
• Although the project agreement provides the responsible entity with a mechanism to address non-performance, payments under the agreement are not subject to reduction for non-performance as asserted in the Project Report. (View report summary)
• A lack of public reporting since the Project Report prevents external stakeholders such as government or taxpayers from assessing the results of the project against their expectations. (View report summary)
• The responsible entity was unable to provide documentation to support the evaluation for two financially significant contract amendments. This documentation is necessary to preserve key knowledge and information. (View report summary)
• The soundness of the sole tenderer’s financing plan has not been demonstrated. (View report summary)
• Despite a higher-than-expected public financial participation, the financial model remains “fragile.” The large uncertainty that surrounds the key assumptions and the sensitivity of the project’s cost-effectiveness to small variations in those assumptions explain this “fragility.” (View report summary)
• The Design-Build-Finance-Maintain contract was awarded to the consortium whose proposal provided the lowest net present value of life cycle costs based on the specified standards over the project’s timeframe—both as compared to the other proposals received, and as compared to the Public Sector Comparator. (View report summary)
• The audit team did not find evidence that estimated risk costs were, in total, validated against actual experience from prior school construction projects. (View report summary)
• A Value for Money Report was not published in accordance with the procurement framework guidance. The Departments did not demonstrate, in a transparent manner, how value for money was obtained. (View report summary)
• The procurement was conducted in a fair and open manner. (View report summary)
• The audit team concluded that the preferred SkyTrain option is likely the best one to meet government’s objectives. However, this conclusion relied on information that was not presented or adequately explained in the submission to Treasury Board. Getting this right despite the information shortfalls is not a cause for complacency. Relying on the same approach in future capital asset projects puts government at risk of making decisions that would have been modified had government understood the full costs, benefits and risks. (View report summary)

Examples of Recommendations

• For future major projects, the Government should establish a senior project oversight committee early in the planning phase of a project. This committee, composed of individuals with considerable experience in managing major projects, should provide advice to the Government and, where relevant, the Legislative Assembly, on the steps required to develop a major project, and should act as a forum for discussing project objectives, risks, procurement, and other relevant matters. (View report summary)
• The Department should conduct a preliminary assessment to identify the best procurement approach prior to a Cabinet decision on how to proceed (P3 or traditional approach). (View report summary)
• The Department should document the development of significant assumptions for the VFM analysis, especially the assessment of their reasonableness. (View report summary)
• The Department should perform a sensitivity analysis which includes all key variables in the project cost estimate process. (View report summary)
• The Department should perform an independent due diligence review of the value for money assessment for each proposed P3 project. (View report summary)
• All documents related to key changes in a P3 Project Agreement should be retained. (View report summary)
• P3 project budgets should include explicit contingency budgets for variations. (View report summary)
• All Project Reports should be reviewed independently before they are publicly released to ensure that key assumptions and disclosures are supported. (View report summary)
• Formal requirements for public reporting should be established after the completion of the capital construction phase and for set times throughout the operational contract. These reports should assess how well the project has achieved its value-for-money/risk transfer objectives in the respective areas. (View report summary)
• The responsible entities should document project reviews so that the scope of these reviews, and the analysis underpinning decisions, are clearly described in written records. (View report summary)
• The Department should ensure the developers maintain adequate documentation to show maintenance work is completed on a timely basis. The Department should review this documentation to ensure maintenance work is completed on a timely basis. (View report summary
• The Department should establish adequate contract management processes to ensure contracted services are received. (View report summary)
• The Department should establish adequate contract management processes to ensure payments made under the P3 contracts comply with contract terms. (View report summary)
•  All significant new contracts between the Department and service providers should include audit provisions for the Province. (View report summary)
•  The Department should define measurable service levels for all services in future contracts and these should be included in the contracts prior to signing. (View report summary)
•  The Department should ensure future contracts describe the contract monitoring process, including documentation requirements and sanctions for instances of non-compliance. (View report summary)

• PreviousProcurement of Goods and Services Using Public-Private Partnerships
• NextProcurement Processes: Oversight Mechanisms and Expertise