RMS justified the decision to adopt the SMC model

RMS prepared a robust business case to justify the decision to use the SMC model to outsource road maintenance in Sydney region south and west zones. RMS also used the business case to secure NSW Government approval to commence the procurement process and to fund the transition costs of outsourcing. The business case addressed how the SMC model would deliver benefits such as cost savings and service delivery improvements. The business case also included estimates of the one-off costs associated with transitioning from in-house provision of road maintenance services, such as redundancy payments.  

Non-financial benefits of the SMC listed in the business case included:

• improved customer service
• improved asset performance
• quality data to enhance decision making
• improved road maintenance delivery
• cultivating innovation.

Road maintenance is a program funded through recurrent, rather than capital funding. While there are no NSW Government guidelines for business cases for these types of programs, RMS broadly complied with NSW Treasury Guidelines for Capital Business Cases (Policy and Guidelines Paper TPP 08–5). At the time, NSW Treasury did not require gateway reviews of business cases for recurrently funded programs.  

In selecting the stewardship contract model, RMS considered experiences with similar arrangements in other jurisdictions, such as New Zealand and the United Kingdom. RMS also reviewed extensive research on outsourcing road maintenance. The SMC model effectively meets RMS’ requirements for flexibility in pricing models, the need for collaboration in asset maintenance planning, promoting innovation and effective performance management.

RMS considered two alternative contracting approaches: alliance contracts (where alliance partners share all outcomes) and traditional contracts (where risk is allocated to each party). Neither of these alternatives met all its requirements.

The SMC performance framework primarily drives effectiveness

The SMC includes a performance framework with mechanisms to regularly assess the contractor’s performance. This framework, based on seven key results areas (KRAs) and 17 key performance indicators (KPIs), is designed to drive contractor performance, with a contractor’s overall margin (profit and overheads) at risk through a risk and reward arrangement. Of the 17 KPIs, 16 are effectiveness measures.

The performance framework is based on the following performance measurement principles:

• alignment with NSW Government and RMS policies such as road quality
• consistent performance measurement across different service providers for different zones
• a risk reward payment mechanism aligned with RMS performance requirements
• contractors encouraged to take a stewardship role over the road network and be involved in defining and delivering good performance
•  be simple and practical and not require undue effort to manage
• flexibility to allow adjustments based on lessons learnt, performance of the contractor and RMS, and changes to NSW Government policy or RMS strategy.

Each KRA has specific KPIs to score the contractor’s performance. Most of the KPIs have objective measures. RMS weighted each KRA and KPI to reflect their relative importance to achieving NSW Government and RMS priorities. While the contractor provides performance data for over half of the KPIs, RMS supplied or verified data impacts on around half of the overall performance score due to the weighting set by RMS.

The SMC duration is separated into three consecutive works periods of one year, three years and three years. The KRAs and KPIs are fixed for each works period, but can be amended by agreement for each new works period. To date, there has been one amendment to make the efficiency KPI more specific. RMS and the contractors are considering other amendments to KPIs for the start of the next works period in July 2018. These include increasing the number of milestones to be measured for the efficiency KPI, and amending the measurement methodology for the KPIs in the asset outcomes KRA.

RMS and the contractor agree on the KPI score against each of the KRAs at the end of each year, resulting in an overall performance score. The SMCs include a performance incentive regime which puts a proportion of the contractor’s overall margin at risk if the contractor does not achieve a specified overall performance score. If the overall performance score exceeds the specified threshold, the contractor can claim bonus payments. To further drive effectiveness, the SMC includes an increase in the specified threshold score at the end of the initial works period.

The SMC pricing mechanisms drive efficiency

The SMC includes three pricing mechanisms that drive efficiency: target cost payments; transferring work to fixed price payment over time; and an efficiency commitment by the contractor to reduce prices by a defined amount over time.  

The first pricing mechanism is target cost. The SMC primarily uses a mix of fixed price and target cost payment types. The payment type used depends on the price predictability of the services that the contractor performed, which can change as the contract matures, or may depend on the type of service.  

Under target cost payment, RMS and the contractor negotiate a target cost for a specific service. If the actual cost exceeds the target cost, RMS pays the contractor the actual cost incurred (including margin), less 50 per cent of the cost overrun. If the actual cost is less than the target cost, then the contractor keeps 25 per cent of any underspend, with another 25 per cent of the underspend transferred into a performance incentive pool. RMS keeps the remaining 50 per cent. RMS can verify the contractor’s actual costs through an 'open book' arrangement, where the contractor gives RMS access to all its cost data.  

The contractor can qualify for additional payments from the incentive pool if its annual overall performance score under the performance framework exceeds the specified threshold. This performance incentive payment can reach a maximum of $1.5 million per year. The amount payable will depend on the total in the incentive pool, and the annual overall performance score the contractor achieves. Conversely, if the contractor fails to achieve the specified threshold overall performance score, then RMS can reduce the contractor’s overall margin for that year by up to 75 per cent. The amount of the reduction will depend on the annual overall performance score the contractor achieves.

The second pricing mechanism is moving more work to fixed price payment over time. RMS can transfer certain types of maintenance work from target cost to fixed price, as the SMC matures and the contractor becomes more familiar with the road network. This has the effect of transferring additional risk to the contractor, thereby promoting efficiency. For the initial 12-month works period, fixed price was only used for program management and asset management services, which represented around eight to nine per cent of the contract value. In the next (three-year) works period, the routine maintenance and pavement rehabilitation categories of road maintenance transitioned from target cost to fixed price, with the fixed price amounts negotiated by RMS and the contractor. This resulted in fixed price payments increasing to between 38 and 44 per cent of contract value. In the final three-year works period, due to start in July 2018, the service categories of simple minor improvement works and event management will also transition from target cost to fixed price.

The final pricing mechanism is an efficiency commitment. The SMC required the contractor to reduce their prices and rates for services by an agreed percentage for each works period. The first of these commercial resets occurred in mid-2015. Also, RMS can benchmark a contractor’s performance against similar services being performed in other zones and in other jurisdictions. Under the SMC, RMS can use the benchmarking when negotiating target costs and when categories of maintenance work are transferred from target cost to fixed price.

RMS should consider whether to incorporate stronger sanctions if the contractor provides incorrect data

RMS uses data provided by the contractor to measure performance and calculate payments to the contractor. The SMC includes a specific sanction if RMS finds that the contractor provided incorrect performance data, but no specific sanction if the contractor provides incorrect financial data. If RMS finds that the contactor provided incorrect performance or financial data, RMS can only recover over-payments which may have been made using the incorrect data.  

In the previous section, we explain how the target cost payment can drive efficiency improvements. This relies on the quality of financial data the contractor provides to RMS. It is important that the RMS considers what additional incentives for the contractor to provide accurate financial data to RMS may be needed, as this can affect both the negotiated target cost and the actual payment for the work.

Financial and performance reporting is based on a mix of RMS and contractor provided data  

The contractor reports to RMS monthly on its progressive KPI performance, and annually on its overall performance score which determines the margin payable. To determine the payments the contractor receives for providing services under the target cost method, RMS can access data supporting the contractor’s target cost estimates and actual costs incurred.

The SMC requires the contractor to have an integrated contract management system that includes operational plans, processes and procedures. The contractor must give RMS access to this system for monitoring and auditing.

The integrated contract management system contains 25 service plans detailing how the contractor will carry out its service delivery obligations. The service plans must use RMS standard specifications for general and technical requirements. The contractor warrants that the service plans comply with RMS requirements and are fit for purpose, and RMS relies on this warranty.

Transparency is one of the principles in the SMC to embed trust and to ensure that RMS remains an informed asset owner. All key contractual information, including financial and performance data, is to be fully transparent and conducted on an 'open book' basis between RMS and the contractor. This is particularly important as contractor provided data impacts on around half of the overall performance score which determines margins and performance incentive payments and on around half of overall contractor payments.  

The SMC enables RMS to effectively audit and oversee contractor performance

The SMC gives RMS the authority to audit and oversee the contractor’s performance, including:

• monthly meetings with the contractor to discuss performance and to reach agreement on KPI scores for the month
• annual performance reviews with the contractor to discuss performance and to reach agreement on the overall performance score
• access to the contractor’s workplaces, information, records and personnel to review, conduct surveillance and audit the contractor’s procedures, activities and compliance with SMC requirements
• auditing, inspecting and carrying out surveillance of the contractor’s:

                     - performance against KPIs and KRAs
                     - compliance with systems, processes and procedures specified in the SMC
                     - records and data to verify accuracy of any data, reports and payment claims provided to RMS.

RMS has not done enough to validate the performance and financial data provided by the contractors  

RMS conducted some limited audits of financial data supporting payments made to the contractor in the first year of the SMC, but no audits of performance data to date. Several financial audits reported that contractor provided data did not meet quality standards. Internal audits of RMS’ contract management practices for the SMC also identified data quality issues. While RMS commenced addressing some data issues, these have yet to be fully resolved. This is important because the data is used to measure performance and calculate contractor payments.

RMS commissioned two internal audits of its SMC contract management practices.

Two key issues identified in the first report (March 2015) were that a contract management framework was yet to be documented, and that RMS should improve its document management practices. During the course of the audit, RMS advised us that it had completed the management actions to address these findings.  

The second report (December 2016) identified the following issues:

• 45 per cent (125 out of 280) of projects were commenced before an agreed target cost had been recorded
• the quality of milestone and financial data in the integrated contract management system, used to prepare monthly reports, was deficient
• there was no clarity regarding what processes had been undertaken by RMS contract managers to independently verify the accuracy of reported KPI scores
• RMS contract and commercial managers did not closely review monthly invoices submitted by contractors to identify anomalies or overcharging.

RMS advised that it plans to address the issues raised in the December 2016 report by 30 June 2017. The outstanding issues from these financial and internal audits puts in doubt the reliability of the data used to measure performance and calculate contractor payments in the first three years of the SMC.

A key finding in the March 2015 report was that RMS did not have a structured process in place on how it would assess performance data. In response, RMS agreed to document and implement a structured process for assessing monthly performance data received from the contractor. However, in line with the findings in the December 2016 internal audit report, we found no evidence that RMS systematically verifies the accuracy of reported KPI scores.

Our analysis of documentation and interviews with RMS contract management staff reinforced many of the findings in the December 2016 internal audit report. While RMS advised it used external reviews to test the validity of progress claims made by the contractor, these reviews ceased after the 2014–15 financial year. RMS also does not routinely interrogate the data that the contractor provides on its monthly performance outcomes. RMS specialist areas provided some level of verification for KPIs dealing with environmental management and customer engagement.

RMS initiated independent reviews to test the veracity of payment claims made by the contractors for projects completed in 2014–15 financial year under the target cost payment mechanism. The reviews sampled claimed reimbursable costs made for 54 south zone projects costing around $30 million, and 53 west zone projects costing around $40 million.  

The west zone review found that nothing had come to the attention of the review to indicate that, in all material respects, actual costs (in total) were not verified. However, the review did raise matters that limited its ability to perform and conclude on certain procedures at transaction detail level, and which raised future risks to project outcomes. These included:

• an inability to verify the accuracy and validity of labour charges recorded in the contractor’s project ledger
• lack of transparency of claimed overhead costs
• inconsistent application of calculations and formulas by the contractor.

The south zone review concluded that it had not been provided with sufficient information to determine that, in all material respects, actual costs were successfully verified for the contractor’s target cost projects. The key issue was the inability of the contractor to provide evidence of actual costs incurred.

In response, RMS advised that the south zone contractor was implementing a new financial management system to address the problems identified. RMS has not indicated when the contractor will complete this implementation.

As both reviews raised issues with the quality of financial data that the contractor provided, together with the findings of the RMS December 2016 internal audit report, we conclude that RMS has yet to effectively resolve the data quality issues first identified in 2015.  

RMS' management of the SMC has key elements missing which reduces its effectiveness

Effective contract management requires a contract management framework that addresses governance arrangements, skills, roles and responsibilities, and policies and procedures. It should promote accountability for decision making and expenditure of public funds. Our research identified nine key elements that we first published as a better practice contract management framework in ‘Managing IT Services contracts’ (February 2012). Details of the better practice contract management framework are in Appendix 2.  

RMS has a contract management framework that includes most elements of the better practice contract management framework, including:

• appropriate reporting and oversight practices are in place
• independent monitoring of contracts to check compliance and identify weaknesses
• roles and responsibilities of RMS and the contractor are clear
• a whole of agency procurement manual with policies and procedures
• established capability to manage the contract.

However, key elements of the better practice contract management framework are missing. Effective contract management should be supported by contract specific procedures, with explanations of, and allocation of responsibility for, the various interventions RMS may be required to exercise in the SMC.  

RMS does not have SMC specific procedures to guide its contract managers.

RMS’ accreditation under the Agency Accreditation Scheme for Construction enables it to carry out most procurement without oversight from the Department of Finance, Services and Innovation. A key condition of accreditation is that it must have contract administration procedures in place.

The SMC is relatively new and very complex and RMS does not provide clear guidance to its contract managers explaining how, when and who can deal with the many mandatory and optional interventions (over 90 in the general conditions alone) RMS may exercise in managing the SMC. There are many other potential interventions in the 24 schedules that make up the SMC. While RMS has implemented a maintenance contract management framework, its framework does not include SMC specific procedures.  

SMC specific contract management procedures may have prevented the following failure by RMS to exercise significant requirements in the SMC. The SMC includes a provision to promote efficiency by requiring the contractor to tender specific percentage reductions in its pricing for services for each of the three-year works period following the initial works period. This is called the 'efficiency commitment'. The contractor needs to demonstrate to RMS how it has accounted for the efficiency commitment when pricing its services for the upcoming three-year works period. This is important because the contractor sets the service pricing at the beginning of each works period.  

However, when pricing services for the first three-year works period, which commenced in July 2015, RMS did not require either contractor to demonstrate how it had accounted for the efficiency commitment. Contrary to the intent of the SMC, RMS advised it will conduct this process at the end of the current works period ending in June 2018. This approach presents a risk that RMS will not know whether the contractor had reduced its prices in line with its efficiency commitment. RMS is waiting three years to find out and will then have to recover any price reductions for that works period.

Having SMC specific procedures in place would also have reduced the risk of RMS not implementing a systematic audit program to validate the data that the contractor provided to support performance outcomes and payment claims.  

In line with the better practice contract management framework, we also expected that RMS would have contract specific allocation of responsibilities to exercise key clauses in the SMC.

RMS provided a copy of its corporate financial delegations, but these were high level and did not reference the SMC. The consequence is that accountability for initiating audits of the data that the contractors provide to RMS is unclear and increases the risk that audits will not be systematically conducted.

There are other areas where RMS should improve its contract management practices. The SMC contractor must prepare and comply with 25 service plans defining required services. The plans include contractor initiated quality assurance audits. While this is good practice, RMS has not systematically reviewed whether either contractor conducted these audits, and whether they had effectively addressed any issues raised.

The annual report, prepared by the contractor, is a key document which supports RMS’ annual review of individual KPIs and the overall performance scores. The monthly report, also prepared by the contractor, acts as a record of agreement between the contractor and RMS on the progressive assessment of KPI scores. We found that some reports contained obvious inaccuracies or omissions, and RMS had not validated the performance information reported, despite endorsement by RMS’ contract manager.  

RMS has not done enough to validate overall performance scores

Under the performance framework, RMS and the contractor agree on the overall performance score at the end of each year financial year. The SMC includes a performance incentive regime which puts up to 75 per cent of the contractor’s overall margin at risk if the contractor does not achieve a specified overall performance score. If the overall performance score exceeds the specified threshold, RMS can make bonus payments. To further drive effectiveness, the threshold score for bonus payments increased at the end of the initial works period.  

For the at-risk margin, RMS pays the full margin if the overall performance score reaches the threshold of 70 points or more. If the score is below 30, then RMS does not pay 75 per cent of the margin. Scores between 30 and 70 result in a pro-rata adjustment to the at-risk margin. To promote performance improvements over time, the threshold score was set at 70 in year one, and 75 in subsequent years.

In each of the two completed years of the SMC, both contractors achieved overall performance scores greater than the specified thresholds at which their margins are reduced.

Despite the overall performance score results, the contractors have reported, and addressed performance deficiencies progressively as they occurred. This is in line with the way RMS incorporated performance incentives in the SMCs. These incentives act to encourage the contractor to report, and respond quickly to, performance shortcomings, minimising the need for RMS to intervene.

However, the underlying risk remains that the overall performance score is only as good as the quality of the data used to calculate the individual KPI scores. With the issues around the quality of contractor supplied data being unresolved, RMS cannot assure us that a contractor’s overall performance scores are robust.

RMS achieved benefits in the first year, but has not tracked benefits since then

A benchmarking study that RMS commissioned for the 2014–15 financial year showed that it had achieved cost savings for service delivery in the first year, and these were around 80 per cent of cost savings estimated by RMS following the completion of the tender process. The SMCs had achieved a 5.5 per cent recurrent cost reduction compared to expected net savings of 6.9 per cent. RMS has not tracked benefits since then.

In calculating the savings for direct cost items, the study acknowledged that there were gaps in available data which resulted in elements of work either being excluded, or relying on data sampling.  

The study was not able to assess the non-cost benefits outlined in the business case, as RMS had not establish a comparative baseline pre-SMC. These benefits were:

• improved customer service
• improved asset performance
• quality data to enhance decision making
• improved road maintenance delivery
• cultivating innovation.

The study also did not compare the actual one-off transition costs to those outlined in the business case.  

In the business case, RMS proposed to monitor success and assign ownership of the benefits. This included establishing a benefits tracking tool. To date, RMS has not implemented the proposed benefits tracking tool.  

The benchmarking study also reviewed whether the SMC effectively achieved:

•  innovation and change implementation
• information availability and accuracy
• Transport Management Centre satisfaction (RMS client)
• Journey Management satisfaction (RMS client)
• KPI performance.

Overall, the study found that the SMC delivered improved performance. However, the improvements related to either the contract structure (having a comprehensive set of KPIs) and improvements on performance from the start of the SMC to the time of the study, rather than being compared to the previous operating model. The only KPI the review was able to compare to pre-SMC was for safety. This showed marked improvement in the total recordable injury frequency rate by the two zone contractors in 2014–15, when compared to not only Sydney region, but also to Hunter and Southern regions on RMS performed road maintenance in 2012–13.

During the course of the audit, RMS advised that it intends to repeat this type of study to determine whether it has achieved all expected benefits (and their value), and that it would use the results to inform its negotiation with the SMC contractors as part of the commercial reset due in mid-2018.