Sydney Road Maintenance Contracts

Overview

In November 2013, Roads and Maritime Services (RMS) outsourced the maintenance of State roads in the Sydney region south and west zones using an innovative contracting approach called the Stewardship Maintenance Contract (SMC). The SMC links risk to reward, and uses a performance framework where outcomes should drive improved performance over time.

RMS’ SMC contract management includes most elements of good practice, including governance and dispute resolution mechanisms. However, key elements are missing which reduces its effectiveness.

1. Executive Summary

Roads and Maritime Services (RMS) is responsible for the Sydney region State roads network This includes over 2,800 kilometres of roads and associated road corridor infrastructure such as bridges, tunnels and drainage structures. RMS divides the network into three geographical areas: south, west and north zones.

In 1995, RMS first outsourced road corridor infrastructure maintenance for the north zone through a Performance Specified Maintenance Contract (PSMC). The current 10-year PSMC for the north zone will expire in October 2018. Prior to November 2013, RMS maintained roads in the south and west zones through its Road and Fleet Services unit. 

In November 2013, RMS outsourced road maintenance services for the south and west zones using Stewardship Maintenance Contracts (SMC). The contracts run for seven years with an option for a further three years at RMS’ discretion. RMS estimated that the annual cost of these contracts was around $240 million in total. In March 2018, the contract prices are due to be reset by negotiation to reflect the contractors’ experience with, and better information about, the road networks and routine maintenance requirements. 

The SMC model adopted stewardship principles to improve value for money. RMS defined stewardship principles as a broad set of values, attitudes and behaviours, required of the contractor to effectively manage the assets on behalf of RMS. The SMC also includes commercial principles, such as linking risk to reward, and a performance framework where outcomes drive performance.

This audit assessed whether RMS had effectively managed the outsourcing of road maintenance in the Sydney region south and west zones. In making this assessment, we answered the following questions:

  1. Did RMS justify the decision to adopt the SMC model?
  2. Do SMCs include key performance indicators (KPIs) and incentives which promote efficiency and effectiveness? 
  3. Does RMS collect high quality information on contractor performance and take action to correct performance deficiencies?
  4. Are the expected benefits being achieved?

Conclusion

RMS developed an innovative contracting approach with the SMC. RMS has realised some benefits in the first year, including savings, from outsourcing road maintenance in the Sydney region south and west zones using the SMC. However, RMS’ management of the SMC has key elements missing which reduces its effectiveness.

The SMC includes performance measures and incentives to drive efficiency and effectiveness improvements over time.  

RMS has established a contract management framework which includes most elements of good practice, including governance and dispute resolution mechanisms. However, it does not have procedures to guide its contract managers in managing specific provisions of the SMC. Consequently, RMS has not exercised several significant SMC requirements, such as having the contractor account for an efficiency dividend in its pricing at the start of each three-year works period. It also has not done enough to assure itself that the contractor provided performance and financial data are correct. This is important because the data is used to measure performance and calculate contractor payments.  

RMS assessed that it had achieved around 80 per cent of the expected cost benefit in the initial year of the SMC. However, it has not tracked its achievement of benefits since then.

The Stewardship Maintenance Contract

RMS justified adopting the SMC model and included KPIs to drive efficiency and effectiveness

The SMC model includes features that RMS had not previously used for road maintenance contracts. These included adopting stewardship principles and transferring price risk to the contractor over time as the contractor becomes familiar with the assets being maintained.

The SMC model meets RMS’ requirements for flexibility in pricing models, the need for collaboration in asset maintenance planning, promoting innovation and effective performance management.

RMS used many good practices to develop the SMC model, including:

  • preparing a robust business case comparing the SMC model to RMS maintaining the road network itself, as well as assessing whether two other contracting models
    (traditional and alliance) would meet its requirements
  • assessing experiences with similar arrangements in other jurisdictions and identifying elements that worked to get the best outcomes
  • developing a robust performance framework, which included a mix of efficiency and effectiveness KPIs that reflected NSW Government policy and RMS priorities
  • incorporating risk and reward incentives delivered through cost sharing arrangements which change as the contract matures
  • using a contract duration that supports RMS priorities and provides an incentive for better quality outcomes.

RMS uses data provided by the contractor to measure performance and calculate payments to the contractor. The SMC includes a specific sanction if RMS finds that the contractor provided incorrect performance data, but no specific sanction if the contractor provides incorrect financial data. If RMS finds that the contactor provided incorrect performance or financial data, RMS can only recover over-payments which may have been made using the incorrect data.  

To provide a stronger incentive for the contractor to ensure data it provides is accurate, RMS should consider whether to incorporate stronger sanctions when negotiating the commercial reset due in mid-2018 for south and west zones. RMS should also consider this for the new contract for the north zone when the current PSMC contract expires in October 2018.

RMS' contract management approach and benefits realization

RMS can improve the effectiveness of its oversight and management of the SMC

RMS does not have SMC specific contract procedures to guide its contract managers. Consequently, RMS has not exercised several significant SMC requirements, such as having the contractors account for an efficiency dividend in their pricing at the start of each three-year works period. Effective contract management should be supported by contract specific procedures, with explanations of, and allocation of responsibility for, the various interventions that RMS may be required to exercise in the SMC.

Performance and financial reporting under the SMC is based on a mix of RMS and contractor provided data. While there are a range of audits of contractor provided performance and financial data that RMS can conduct each year under the SMC, it does not have a schedule of audits it will conduct and when.  
During the first year of the SMC, RMS commissioned some limited audits of financial data. In the first three years of the SMC, RMS did not conduct any audits of performance data. Had there been SMC specific procedures in place, this would have reduced the risk of RMS not implementing a systematic audit program to give it reasonable assurance on the quality of the data that the contractor has provided. This is important because the data is used to measure performance and calculate contractor payments.

RMS has been aware of data quality issues since 2015. While RMS advised that it commenced addressing some data quality issues in response to a series of reviews conducted in 2015, a recent internal audit report indicates that RMS has not resolved the data quality issues.  

RMS achieved benefits in the first year, but has not tracked benefits since

As part of the business case, RMS agreed to implement a benefits realisation strategy, including a benefits tracking tool. RMS commenced tracking benefits, but did not establish a comparative baseline pre-SMC on non-financial benefits, and has not tracked benefits past year one.

In 2015, a benchmarking study commissioned by RMS found that it had achieved 80 per cent of the expected recurrent cost savings and other benefits, such as improved workplace safety, in the first full year of the SMC. However, there was no clear baseline to measure
non-financial performance. The study was qualified due to gaps in available data. The study also did not reconcile the actual one-off transition costs to the business case estimate.

During the course of the audit, RMS advised that it intends to repeat this type of study to determine whether it has achieved all expected benefits (and their value), and that it would use the results to inform its negotiation with the SMC contractors as part of the commercial reset due in mid-2018.

2. Introduction

Roads and Maritime Services is responsible for the State Roads network in the Sydney region

Roads and Maritime Services (RMS) is responsible for the Sydney region State roads network. This includes over 2,800 kilometres of roads and associated road corridor infrastructure such as bridges, tunnels and drainage structures. The network is divided into three geographical areas: south, west and north zones. Prior to November 2013, RMS maintained roads in the Sydney region south and west zones through its Road and Fleet Services unit.  

In 1995, RMS first outsourced road corridor infrastructure maintenance for the north zone through a Performance Specified Maintenance Contract (PSMC). The current 10-year PSMC for the north zone will expire in October 2018. This contract is worth around $35 million per annum.  

NSW Government priorities and road maintenance

Efficient and effective road maintenance contributes to the following NSW Government priorities:

  • improving road travel reliability
  • ensuring on-time running of public transport
  • reducing road fatalities
  • improving government services
  • keeping our environment clean.

The NSW Commission of Audit recommended outsourcing the maintenance of State roads

The NSW Commission of Audit in its Final Report on Government Expenditure (May 2012) recommended contestability as an appropriate strategy to consider for improving road maintenance service delivery for State roads.  

The Commission benchmarked RMS’ road surface quality and cost per lane kilometre against those of Western Australia, Victoria, and Queensland. This showed that New South Wales lagged the other states on both these measures.  

Exhibit 1: Interjurisdictional comparison of road maintenance outcomes 2009–10
  WA VIC QLD NSW
Roads managed (lane kms) 52,659 50,510 71,353 80,348
Estimated spend ($/lane km) 5,000 4,500 6,000 7,000
Road quality measure (%) 99 99 94 91

Source: NSW Commission of Audit Final Report May 2012.

The Commission noted that RMS had conducted two independent reviews to examine the potential for extending road maintenance contestability. The Commission found that there was inadequate and inconclusive benchmarking to establish the efficiency of RMS’ Road and Fleet Services unit when compared to outsourcing. It recommended that RMS bring forward a proposal to conduct a competitive tender for the road maintenance of the Sydney region south zone road network to inform the feasibility of a progressive rollout of road maintenance contestability across other areas of the State. In August 2012, the NSW Government adopted the Commission’s recommendation.

The NSW Government introduced road maintenance contestability through Stewardship Maintenance Contracts

In April 2013, the NSW Government announced that it would introduce road maintenance contestability across the Sydney region, using a Stewardship Maintenance Contract (SMC) model to improve value for money. In doing so, it excluded RMS’ Road and Fleet Services unit from tendering.  

The SMC model is based on the following key commercial and performance principles set by RMS:

  • performance driven by outcomes
  • flexible and adaptable
  • transparent and measurable
  • linking risk to reward
  • continuous improvement
  • criteria for selection of, and transition to, different payment models.

The following key stewardship principles underpin the SMC’s broad set of values, attitudes and behaviours, which are required of the contractor to effectively manage the assets on behalf of RMS:

  • putting RMS’ customers (road users and the general public) first and being responsive to them
  • being responsible and accountable for the outcomes resulting from the management of the assets
  • managing the assets diligently, efficiently and effectively with limited direction from RMS
  • working collaboratively with RMS to deliver services that are tailored to meet RMS’ evolving needs
  • acting with integrity and transparency in performing the services
  • performing the services in the best interests of RMS and asset users.

Other key features of the SMC include:

  • service requirements which describe the scope of the services, and the standards the contractor must meet
  • a commercial framework which defines how payments are structured, how performance assessment will impact on payments and outlines the key commercial principles. SMCs primarily divide payments into two main mechanisms, these being the priced component (or fixed price) and the target cost calculated as follows:
    • fixed price – the contractor is paid a pre-agreed amount for specific services being provided, regardless of the actual costs incurred
    • target cost – RMS and the contractor agree on a target cost for a project, and any cost overruns or underruns are shared between them
  • a performance framework which provides mechanisms for assessing contractor performance. This includes a comprehensive listing of the key result areas (KRAs) and key performance indicators (KPIs) against which RMS measures the contractor’s performance. The framework also outlines the scoring methodology that RMS uses to determine whether the contractor’s bid margin (profit and overheads) is reduced due to less than satisfactory performance or whether a bonus is paid if a threshold performance score is exceeded.

Road maintenance under SMCs for Sydney region south and west zones commenced in November 2013

In November 2013, RMS awarded SMCs to the Leighton Boral Amey consortium, now named Ventia Boral Amey (VBA), for the south zone and the DownerMouchel (DM) consortium for the west zone. The contracts run for seven years with an option for a further three years at RMS’ discretion. In April 2014, full services commenced following a four-month transition period. RMS estimated that the annual cost of these contracts was around $240 million in total. In March 2018, the contract prices are due to be reset by negotiation to reflect the contractors’ experience with, and better information about, the road networks and routine maintenance requirements. 

This image shows that the SMC west zone (Sydney West) contract is owned by DownerMouchel, the PSMC North Zone (Sydney North) by Downer EDI works and the SMC South Zone (Sydney South) by Leighton Boral Amey
Exhibit 2: Road maintenance contracts in the Sydney region
Source: Roads and Maritime Services 2015. 

Audit scope and focus:

This audit assessed whether RMS had effectively managed the outsourcing of road maintenance in the Sydney region south and west zones. In making this assessment, we answered the following questions:

Audit question What the audit examined
Did RMS justify the decision to adopt the SMC model? We looked at how RMS reached the decision that SMCs were the most appropriate contracting model for outsourcing road maintenance in the Sydney region south and west zones. This included whether the RMS business case supporting the adoption of the SMC model was consistent with NSW Treasury guidelines.
We also considered whether the KPIs and incentives in the SMC reflected NSW Government and RMS priorities.
Do SMCs include performance indicators (KPIs) and incentives which promote efficiency and effectiveness? We looked at whether the performance indicators and incentives worked to reduce costs, improve the way road maintenance services were delivered, and improve road quality.
Does RMS collect high quality information on contractor performance and take action to correct performance deficiencies? We looked at what assurance RMS had that contractor provided performance and financial data was accurate. We also looked at what action RMS took if it finds performance deficiencies.
Are the expected benefits being achieved? We sought to determine whether RMS could demonstrate that it was achieving the benefits outlined in the business case for adopting the SMC model.

 

3. Recommendations

  1. Roads and Maritime Services should consider whether to incorporate stronger sanctions in the Stewardship Maintenance Contract if the contractor provides incorrect performance or financial data to RMS, when:
     
    1. negotiating the commercial reset for the next works period with the Sydney region south and west zone contractors due in July 2018.
    2. finalising a new SMC contract for the Sydney region north zone, due to commence in October 2018.

Roads and Maritime Services should, by September 2017:

2.  Review its contract management framework for SMCs to ensure that all authorities and accountabilities of
     contract managers are clearly defined, including:

a) accountability and procedures for exercising all operational clauses in the SMC where RMS may opt to, or be required to intervene, or make a decision

b) authority to approve or initiate the interventions RMS is required to, or may, exercise under the SMC

c) the audits that RMS will conduct to systematically validate the performance and financial data that the SMC contractors provide

d) the accountabilities of RMS contract managers to systematically review audits and quality reviews that the SMC contractors must conduct to demonstrate compliance with their service plans

e) the accountabilities of RMS contract managers to check that the monthly and annual reports provided by SMC contractors do not contain errors, omissions or inaccuracies.

3.  Improve its management of benefits realisation by:

a) initiating a further benefits realisation review and record the benefits delivered against those
    estimated following the tender process, including the one-off transition costs

b) identify any benefits, including savings, not yet attained and develop strategies to address any short-falls

c) establish a tool to track the ongoing realisation of benefits.

4. Key Findings

4.1 The Stewardship Maintenance Contract

RMS developed an innovative contracting approach with the SMC. The SMC model includes features that RMS had not previously used for road maintenance contracts. These included adopting stewardship principles and transferring price risk to the contractor over time as the contractor becomes familiar with the assets being maintained.

The SMC model meets RMS’ requirements for flexibility in pricing models, the need for collaboration in asset maintenance planning, promoting innovation and effective performance management.

RMS used many good practices to develop the SMC model, including:

  • preparing a robust business case comparing the SMC model to RMS maintaining the road network itself, as well as assessing whether two other contracting models (traditional and alliance) would meet its requirements
  • assessing experiences with similar arrangements in other jurisdictions and identifying elements that worked to get the best outcomes
  • incorporating a robust performance framework, which included a mix of efficiency and effectiveness KPIs that reflected NSW Government policy and RMS priorities
    incorporating risk and reward incentives delivered through cost sharing arrangements to promote efficiency
  • using a contract duration that supports RMS priorities and provides an incentive for better quality outcomes.

RMS uses data provided by the contractor to measure performance and calculate payments to the contractor. The SMC includes a specific sanction if RMS finds that the contractor provided incorrect performance data, but no specific sanction if the contractor provides incorrect financial data. If RMS finds that the contactor provided incorrect performance or financial data, RMS can only recover over-payments which may have been made using the incorrect data.  

To provide a stronger incentive for the contractor to ensure data it provides is accurate, RMS should consider strengthening the sanctions when negotiating the commercial reset due in mid-2018 for south and west zones. RMS should also consider this for the new contract for the north zone when the current PSMC contract expires in October 2018.

Recommendations

1. RMS should consider whether to incorporate stronger sanctions in the SMC if the contractor provides incorrect performance or financial data to RMS:

a) when negotiating the commercial reset for the next works period with the Sydney region south and west zone contractors due in July 2018

b) when finalising a new SMC contract for the Sydney region north zone, due to commence in October 2018.

4.1.1 The SMC business case

RMS justified the decision to adopt the SMC model

RMS prepared a robust business case to justify the decision to use the SMC model to outsource road maintenance in Sydney region south and west zones. RMS also used the business case to secure NSW Government approval to commence the procurement process and to fund the transition costs of outsourcing. The business case addressed how the SMC model would deliver benefits such as cost savings and service delivery improvements. The business case also included estimates of the one-off costs associated with transitioning from in-house provision of road maintenance services, such as redundancy payments.  

Non-financial benefits of the SMC listed in the business case included:

  • improved customer service
  • improved asset performance
  • quality data to enhance decision making
  • improved road maintenance delivery
  • cultivating innovation.

Road maintenance is a program funded through recurrent, rather than capital funding. While there are no NSW Government guidelines for business cases for these types of programs, RMS broadly complied with NSW Treasury Guidelines for Capital Business Cases (Policy and Guidelines Paper TPP 08–5). At the time, NSW Treasury did not require gateway reviews of business cases for recurrently funded programs.  

In selecting the stewardship contract model, RMS considered experiences with similar arrangements in other jurisdictions, such as New Zealand and the United Kingdom. RMS also reviewed extensive research on outsourcing road maintenance. The SMC model effectively meets RMS’ requirements for flexibility in pricing models, the need for collaboration in asset maintenance planning, promoting innovation and effective performance management.

RMS considered two alternative contracting approaches: alliance contracts (where alliance partners share all outcomes) and traditional contracts (where risk is allocated to each party). Neither of these alternatives met all its requirements.

4.1.2 The SMC includes performance measures and incentives to drive efficiency and effectiveness with opportunity for improvement

The SMC performance framework primarily drives effectiveness

The SMC includes a performance framework with mechanisms to regularly assess the contractor’s performance. This framework, based on seven key results areas (KRAs) and 17 key performance indicators (KPIs), is designed to drive contractor performance, with a contractor’s overall margin (profit and overheads) at risk through a risk and reward arrangement. Of the 17 KPIs, 16 are effectiveness measures.

The performance framework is based on the following performance measurement principles:

  • alignment with NSW Government and RMS policies such as road quality
  • consistent performance measurement across different service providers for different zones
  • a risk reward payment mechanism aligned with RMS performance requirements
  • contractors encouraged to take a stewardship role over the road network and be involved in defining and delivering good performance
  •  be simple and practical and not require undue effort to manage
  • flexibility to allow adjustments based on lessons learnt, performance of the contractor and RMS, and changes to NSW Government policy or RMS strategy.

Each KRA has specific KPIs to score the contractor’s performance. Most of the KPIs have objective measures. RMS weighted each KRA and KPI to reflect their relative importance to achieving NSW Government and RMS priorities. While the contractor provides performance data for over half of the KPIs, RMS supplied or verified data impacts on around half of the overall performance score due to the weighting set by RMS.

Exhibit 3: Stewardship Maintenance Contract Key Result Areas and Key Performance Indicators
KRA Max score KPI Weight % Measures > Source of data
Stewardship 5 1. Overall performance -- Subjective / Effectiveness >Joint
    2. Compliance with Road Occupancy Licence -- Objective / Effectiveness >Contractor
    3. Response to natural disasters and weather events -- Objective / Effectiveness >Contractor
    4. Response to safety hazards -- Objective / Effectiveness >Contractor
Customer experience 15 5. Customer engagement and consultation 40 Objective / Effectiveness >Contractor
    6. Customer complaints made 30 Objective / Effectiveness >Contractor
    7. Response to customer complaints 30 Objective / Effectiveness >Contractor
Network outcomes 10 8. Incident management 100 Objective / Effectiveness >Contractor
Environment 5 9. Environment management 100 Objective / Effectiveness >RMS
Asset outcomes 40 10. Smooth travel exposure 35 Objective / Effectiveness >RMS
    11. Road surface cracking 20 Objective / Effectiveness >RMS
    12. Road surface rutting 20 Objective / Effectiveness >RMS
    13. Skid resistance 25 Objective / Effectiveness >RMS
Program governance 20 14. Compliance with contract requirements 40 Objective / Effectiveness >RMS
    15. Relationship 30 Subjective / Effectiveness >Joint
    16. Safety   Subjective and objective /
Effectiveness >Contractor
Efficency 10 17. Time performance of projects 100 Objective / Efficiency >Contractor

Source: Roads and Maritime Services 2013.

The SMC duration is separated into three consecutive works periods of one year, three years and three years. The KRAs and KPIs are fixed for each works period, but can be amended by agreement for each new works period. To date, there has been one amendment to make the efficiency KPI more specific. RMS and the contractors are considering other amendments to KPIs for the start of the next works period in July 2018. These include increasing the number of milestones to be measured for the efficiency KPI, and amending the measurement methodology for the KPIs in the asset outcomes KRA.

RMS and the contractor agree on the KPI score against each of the KRAs at the end of each year, resulting in an overall performance score. The SMCs include a performance incentive regime which puts a proportion of the contractor’s overall margin at risk if the contractor does not achieve a specified overall performance score. If the overall performance score exceeds the specified threshold, the contractor can claim bonus payments. To further drive effectiveness, the SMC includes an increase in the specified threshold score at the end of the initial works period.

The SMC pricing mechanisms drive efficiency

The SMC includes three pricing mechanisms that drive efficiency: target cost payments; transferring work to fixed price payment over time; and an efficiency commitment by the contractor to reduce prices by a defined amount over time.  

The first pricing mechanism is target cost. The SMC primarily uses a mix of fixed price and target cost payment types. The payment type used depends on the price predictability of the services that the contractor performed, which can change as the contract matures, or may depend on the type of service.  

Under target cost payment, RMS and the contractor negotiate a target cost for a specific service. If the actual cost exceeds the target cost, RMS pays the contractor the actual cost incurred (including margin), less 50 per cent of the cost overrun. If the actual cost is less than the target cost, then the contractor keeps 25 per cent of any underspend, with another 25 per cent of the underspend transferred into a performance incentive pool. RMS keeps the remaining 50 per cent. RMS can verify the contractor’s actual costs through an 'open book' arrangement, where the contractor gives RMS access to all its cost data.  

The contractor can qualify for additional payments from the incentive pool if its annual overall performance score under the performance framework exceeds the specified threshold. This performance incentive payment can reach a maximum of $1.5 million per year. The amount payable will depend on the total in the incentive pool, and the annual overall performance score the contractor achieves. Conversely, if the contractor fails to achieve the specified threshold overall performance score, then RMS can reduce the contractor’s overall margin for that year by up to 75 per cent. The amount of the reduction will depend on the annual overall performance score the contractor achieves.

Exhibit 4: Target cost payments - benefits and risks

The target cost payment method includes risk and reward sharing to encourage greater efficiency by a contractor on delivering services. The contractor’s ability to retain a proportion of any underspend of the target cost, or needing to meet a portion of overspend, can act as a strong incentive for the contractor to drive costs out of providing services.

In the SMC, RMS applies this payment method in situations where the scope of services is relatively well defined, but there are risks which can be best managed by the contractor, and/or there are risks which would attract a premium if payment was on a fixed price basis.

However, this payment method also brings significant risks to RMS if not managed well. Firstly, it can be reasonably expected that a contractor will seek to negotiate the highest possible target cost for the service. This will create the greatest potential for underspending and hence retention of a portion of that underspend, or provide an inflated buffer with which the contractor can mitigate losses due to overspending. To avoid this outcome, RMS needs to have adequate numbers of skilled cost estimators who can negotiate target costs for services from a strong knowledge and experience base. Otherwise, the potential efficiency benefit could be lost.

Secondly, the target cost payment method is based on reimbursing the contractor for actual costs incurred (subject to risk/reward additions or subtractions). As a result, any costs the contractor claims which are not genuinely attributable to the specific service will provide an improper benefit to the contractor. While the SMC requires the contractor to provide RMS with complete access to all financial information, RMS must ensure that it verifies the claimed actual costs to avoid being overcharged.

Source: Audit Office research 2017.

The second pricing mechanism is moving more work to fixed price payment over time. RMS can transfer certain types of maintenance work from target cost to fixed price, as the SMC matures and the contractor becomes more familiar with the road network. This has the effect of transferring additional risk to the contractor, thereby promoting efficiency. For the initial 12-month works period, fixed price was only used for program management and asset management services, which represented around eight to nine per cent of the contract value. In the next (three-year) works period, the routine maintenance and pavement rehabilitation categories of road maintenance transitioned from target cost to fixed price, with the fixed price amounts negotiated by RMS and the contractor. This resulted in fixed price payments increasing to between 38 and 44 per cent of contract value. In the final three-year works period, due to start in July 2018, the service categories of simple minor improvement works and event management will also transition from target cost to fixed price.

The final pricing mechanism is an efficiency commitment. The SMC required the contractor to reduce their prices and rates for services by an agreed percentage for each works period. The first of these commercial resets occurred in mid-2015. Also, RMS can benchmark a contractor’s performance against similar services being performed in other zones and in other jurisdictions. Under the SMC, RMS can use the benchmarking when negotiating target costs and when categories of maintenance work are transferred from target cost to fixed price.

RMS should consider whether to incorporate stronger sanctions if the contractor provides incorrect data

RMS uses data provided by the contractor to measure performance and calculate payments to the contractor. The SMC includes a specific sanction if RMS finds that the contractor provided incorrect performance data, but no specific sanction if the contractor provides incorrect financial data. If RMS finds that the contactor provided incorrect performance or financial data, RMS can only recover over-payments which may have been made using the incorrect data.  

In the previous section, we explain how the target cost payment can drive efficiency improvements. This relies on the quality of financial data the contractor provides to RMS. It is important that the RMS considers what additional incentives for the contractor to provide accurate financial data to RMS may be needed, as this can affect both the negotiated target cost and the actual payment for the work.

4.2 RMS' contract management approach and benefits realisation

RMS has established a contract management framework which includes most elements of good practice, including governance and dispute resolution mechanisms. However, RMS' management of the SMC has key elements missing which reduces its effectiveness. RMS realised some benefits in the first year of the SMC, including savings in service delivery costs. However, RMS has not tracked its achievement of benefits since then.

RMS does not have SMC specific contract procedures to guide its contract managers. Consequently, RMS has not exercised several significant SMC requirements, such as having the contractors account for an efficiency dividend in their pricing at the start of each three-year works period. Effective contract management should be supported by contract specific procedures, with explanations of, and allocation of responsibility for, the various interventions that RMS may be required to exercise in the SMC.

Performance and financial reporting under the SMC is based on a mix of RMS and contractor provided data. While there are a range of audits of the contractor provided performance and financial data that RMS can conduct each year under the SMC, it does not have a schedule of audits it will conduct and when. During the first year of the SMC, RMS commissioned some limited audits of financial data. In the first three years of the SMC, RMS did not conduct any audits of performance data. Had there been SMC specific procedures in place, this would have reduced the risk of RMS not implementing a systematic audit program to give it reasonable assurance on the quality of the data that the contractor has provided. This is important because the data is used to measure performance and calculate contractor payments. RMS has been aware of data quality issues since 2015. While RMS advised that it commenced addressing some data quality issues in response to a series of reviews conducted in 2015, a recent internal audit report indicates that RMS has not resolved the data quality issues.  

In 2015, a benchmarking study commissioned by RMS found that it had achieved 80 per cent of the expected recurrent cost savings and other benefits in the first full year of the SMC. However, there was no clear baseline to measure non-financial performance. During the course of the audit, RMS advised that it intends to repeat this type of study to determine whether it has achieved all expected benefits (and their value), and that it would use the results to inform its negotiation with the SMC contractors as part of the commercial reset due in mid-2018.

Recommendations

RMS should, by September 2017:

2. review its contract management framework for SMCs to ensure that all authorities and accountabilities of contract managers are clearly defined, including:

a) accountability and procedures for exercising all operational clauses in the SMC where RMS may opt to, or be required to intervene, or make a decision

b) authority to approve or initiate the interventions RMS is required to, or may, exercise under the SMC

c) the audits that RMS will conduct to systematically validate the performance and financial data that the SMC contractors provide

d) the accountabilities of RMS contract managers to systematically review audits and quality reviews that the SMC contractors must conduct to demonstrate compliance with their service plans

e) the accountabilities of RMS contract managers to check that the monthly and annual reports provided by SMC contractors do not contain errors, omissions or inaccuracies.

3. improve its management of benefits realisation by:

a) initiating a further benefits realisation review and record the benefits delivered against those estimated following the tender process, including one-off transition costs

b) identify any benefits, including savings, not yet attained and develop strategies to address the shortfalls

c) establish a tool to track the ongoing realisation of benefits.

4.2.1 Contract performance and financial monitoring

Financial and performance reporting is based on a mix of RMS and contractor provided data  

The contractor reports to RMS monthly on its progressive KPI performance, and annually on its overall performance score which determines the margin payable. To determine the payments the contractor receives for providing services under the target cost method, RMS can access data supporting the contractor’s target cost estimates and actual costs incurred.

The SMC requires the contractor to have an integrated contract management system that includes operational plans, processes and procedures. The contractor must give RMS access to this system for monitoring and auditing.

The integrated contract management system contains 25 service plans detailing how the contractor will carry out its service delivery obligations. The service plans must use RMS standard specifications for general and technical requirements. The contractor warrants that the service plans comply with RMS requirements and are fit for purpose, and RMS relies on this warranty.

Transparency is one of the principles in the SMC to embed trust and to ensure that RMS remains an informed asset owner. All key contractual information, including financial and performance data, is to be fully transparent and conducted on an 'open book' basis between RMS and the contractor. This is particularly important as contractor provided data impacts on around half of the overall performance score which determines margins and performance incentive payments and on around half of overall contractor payments.  

The SMC enables RMS to effectively audit and oversee contractor performance

The SMC gives RMS the authority to audit and oversee the contractor’s performance, including:

  • monthly meetings with the contractor to discuss performance and to reach agreement on KPI scores for the month
  • annual performance reviews with the contractor to discuss performance and to reach agreement on the overall performance score
  • access to the contractor’s workplaces, information, records and personnel to review, conduct surveillance and audit the contractor’s procedures, activities and compliance with SMC requirements
  • auditing, inspecting and carrying out surveillance of the contractor’s:

                     - performance against KPIs and KRAs
                     - compliance with systems, processes and procedures specified in the SMC
                     - records and data to verify accuracy of any data, reports and payment claims provided to RMS.

RMS has not done enough to validate the performance and financial data provided by the contractors  

RMS conducted some limited audits of financial data supporting payments made to the contractor in the first year of the SMC, but no audits of performance data to date. Several financial audits reported that contractor provided data did not meet quality standards. Internal audits of RMS’ contract management practices for the SMC also identified data quality issues. While RMS commenced addressing some data issues, these have yet to be fully resolved. This is important because the data is used to measure performance and calculate contractor payments.

RMS commissioned two internal audits of its SMC contract management practices.

Two key issues identified in the first report (March 2015) were that a contract management framework was yet to be documented, and that RMS should improve its document management practices. During the course of the audit, RMS advised us that it had completed the management actions to address these findings.  

The second report (December 2016) identified the following issues:

  • 45 per cent (125 out of 280) of projects were commenced before an agreed target cost had been recorded
  • the quality of milestone and financial data in the integrated contract management system, used to prepare monthly reports, was deficient
  • there was no clarity regarding what processes had been undertaken by RMS contract managers to independently verify the accuracy of reported KPI scores
  • RMS contract and commercial managers did not closely review monthly invoices submitted by contractors to identify anomalies or overcharging.

RMS advised that it plans to address the issues raised in the December 2016 report by 30 June 2017. The outstanding issues from these financial and internal audits puts in doubt the reliability of the data used to measure performance and calculate contractor payments in the first three years of the SMC.

A key finding in the March 2015 report was that RMS did not have a structured process in place on how it would assess performance data. In response, RMS agreed to document and implement a structured process for assessing monthly performance data received from the contractor. However, in line with the findings in the December 2016 internal audit report, we found no evidence that RMS systematically verifies the accuracy of reported KPI scores.

Our analysis of documentation and interviews with RMS contract management staff reinforced many of the findings in the December 2016 internal audit report. While RMS advised it used external reviews to test the validity of progress claims made by the contractor, these reviews ceased after the 2014–15 financial year. RMS also does not routinely interrogate the data that the contractor provides on its monthly performance outcomes. RMS specialist areas provided some level of verification for KPIs dealing with environmental management and customer engagement.

RMS initiated independent reviews to test the veracity of payment claims made by the contractors for projects completed in 2014–15 financial year under the target cost payment mechanism. The reviews sampled claimed reimbursable costs made for 54 south zone projects costing around $30 million, and 53 west zone projects costing around $40 million.  

The west zone review found that nothing had come to the attention of the review to indicate that, in all material respects, actual costs (in total) were not verified. However, the review did raise matters that limited its ability to perform and conclude on certain procedures at transaction detail level, and which raised future risks to project outcomes. These included:

  • an inability to verify the accuracy and validity of labour charges recorded in the contractor’s project ledger
  • lack of transparency of claimed overhead costs
  • inconsistent application of calculations and formulas by the contractor.

The south zone review concluded that it had not been provided with sufficient information to determine that, in all material respects, actual costs were successfully verified for the contractor’s target cost projects. The key issue was the inability of the contractor to provide evidence of actual costs incurred.

In response, RMS advised that the south zone contractor was implementing a new financial management system to address the problems identified. RMS has not indicated when the contractor will complete this implementation.

As both reviews raised issues with the quality of financial data that the contractor provided, together with the findings of the RMS December 2016 internal audit report, we conclude that RMS has yet to effectively resolve the data quality issues first identified in 2015.  

4.2.2 RMS' contract management framework

RMS' management of the SMC has key elements missing which reduces its effectiveness

Effective contract management requires a contract management framework that addresses governance arrangements, skills, roles and responsibilities, and policies and procedures. It should promote accountability for decision making and expenditure of public funds. Our research identified nine key elements that we first published as a better practice contract management framework in ‘Managing IT Services contracts’ (February 2012). Details of the better practice contract management framework are in Appendix 2.  

RMS has a contract management framework that includes most elements of the better practice contract management framework, including:

  • appropriate reporting and oversight practices are in place
  • independent monitoring of contracts to check compliance and identify weaknesses
  • roles and responsibilities of RMS and the contractor are clear
  • a whole of agency procurement manual with policies and procedures
  • established capability to manage the contract.

However, key elements of the better practice contract management framework are missing. Effective contract management should be supported by contract specific procedures, with explanations of, and allocation of responsibility for, the various interventions RMS may be required to exercise in the SMC.  

RMS does not have SMC specific procedures to guide its contract managers.

RMS’ accreditation under the Agency Accreditation Scheme for Construction enables it to carry out most procurement without oversight from the Department of Finance, Services and Innovation. A key condition of accreditation is that it must have contract administration procedures in place.

The SMC is relatively new and very complex and RMS does not provide clear guidance to its contract managers explaining how, when and who can deal with the many mandatory and optional interventions (over 90 in the general conditions alone) RMS may exercise in managing the SMC. There are many other potential interventions in the 24 schedules that make up the SMC. While RMS has implemented a maintenance contract management framework, its framework does not include SMC specific procedures.  

SMC specific contract management procedures may have prevented the following failure by RMS to exercise significant requirements in the SMC. The SMC includes a provision to promote efficiency by requiring the contractor to tender specific percentage reductions in its pricing for services for each of the three-year works period following the initial works period. This is called the 'efficiency commitment'. The contractor needs to demonstrate to RMS how it has accounted for the efficiency commitment when pricing its services for the upcoming three-year works period. This is important because the contractor sets the service pricing at the beginning of each works period.  

However, when pricing services for the first three-year works period, which commenced in July 2015, RMS did not require either contractor to demonstrate how it had accounted for the efficiency commitment. Contrary to the intent of the SMC, RMS advised it will conduct this process at the end of the current works period ending in June 2018. This approach presents a risk that RMS will not know whether the contractor had reduced its prices in line with its efficiency commitment. RMS is waiting three years to find out and will then have to recover any price reductions for that works period.

Having SMC specific procedures in place would also have reduced the risk of RMS not implementing a systematic audit program to validate the data that the contractor provided to support performance outcomes and payment claims.  

In line with the better practice contract management framework, we also expected that RMS would have contract specific allocation of responsibilities to exercise key clauses in the SMC.

RMS provided a copy of its corporate financial delegations, but these were high level and did not reference the SMC. The consequence is that accountability for initiating audits of the data that the contractors provide to RMS is unclear and increases the risk that audits will not be systematically conducted.

There are other areas where RMS should improve its contract management practices. The SMC contractor must prepare and comply with 25 service plans defining required services. The plans include contractor initiated quality assurance audits. While this is good practice, RMS has not systematically reviewed whether either contractor conducted these audits, and whether they had effectively addressed any issues raised.

The annual report, prepared by the contractor, is a key document which supports RMS’ annual review of individual KPIs and the overall performance scores. The monthly report, also prepared by the contractor, acts as a record of agreement between the contractor and RMS on the progressive assessment of KPI scores. We found that some reports contained obvious inaccuracies or omissions, and RMS had not validated the performance information reported, despite endorsement by RMS’ contract manager.  

RMS has not done enough to validate overall performance scores

Under the performance framework, RMS and the contractor agree on the overall performance score at the end of each year financial year. The SMC includes a performance incentive regime which puts up to 75 per cent of the contractor’s overall margin at risk if the contractor does not achieve a specified overall performance score. If the overall performance score exceeds the specified threshold, RMS can make bonus payments. To further drive effectiveness, the threshold score for bonus payments increased at the end of the initial works period.  

For the at-risk margin, RMS pays the full margin if the overall performance score reaches the threshold of 70 points or more. If the score is below 30, then RMS does not pay 75 per cent of the margin. Scores between 30 and 70 result in a pro-rata adjustment to the at-risk margin. To promote performance improvements over time, the threshold score was set at 70 in year one, and 75 in subsequent years.

In each of the two completed years of the SMC, both contractors achieved overall performance scores greater than the specified thresholds at which their margins are reduced.

Exhibit 5: Overall performance scores
Year South Zone West Zone Threshold
2014-15 85.61 87.32 70
2015-16 81.39 88.26 75

Source: Roads and Maritime Services 2017.

Despite the overall performance score results, the contractors have reported, and addressed performance deficiencies progressively as they occurred. This is in line with the way RMS incorporated performance incentives in the SMCs. These incentives act to encourage the contractor to report, and respond quickly to, performance shortcomings, minimising the need for RMS to intervene.

However, the underlying risk remains that the overall performance score is only as good as the quality of the data used to calculate the individual KPI scores. With the issues around the quality of contractor supplied data being unresolved, RMS cannot assure us that a contractor’s overall performance scores are robust.

4.2.3 Benefits realisation

RMS achieved benefits in the first year, but has not tracked benefits since then

A benchmarking study that RMS commissioned for the 2014–15 financial year showed that it had achieved cost savings for service delivery in the first year, and these were around 80 per cent of cost savings estimated by RMS following the completion of the tender process. The SMCs had achieved a 5.5 per cent recurrent cost reduction compared to expected net savings of 6.9 per cent. RMS has not tracked benefits since then.

In calculating the savings for direct cost items, the study acknowledged that there were gaps in available data which resulted in elements of work either being excluded, or relying on data sampling.  

The study was not able to assess the non-cost benefits outlined in the business case, as RMS had not establish a comparative baseline pre-SMC. These benefits were:

  • improved customer service
  • improved asset performance
  • quality data to enhance decision making
  • improved road maintenance delivery
  • cultivating innovation.

The study also did not compare the actual one-off transition costs to those outlined in the business case.  

In the business case, RMS proposed to monitor success and assign ownership of the benefits. This included establishing a benefits tracking tool. To date, RMS has not implemented the proposed benefits tracking tool.  

The benchmarking study also reviewed whether the SMC effectively achieved:

  •  innovation and change implementation
  • information availability and accuracy
  • Transport Management Centre satisfaction (RMS client)
  • Journey Management satisfaction (RMS client)
  • KPI performance.

Overall, the study found that the SMC delivered improved performance. However, the improvements related to either the contract structure (having a comprehensive set of KPIs) and improvements on performance from the start of the SMC to the time of the study, rather than being compared to the previous operating model. The only KPI the review was able to compare to pre-SMC was for safety. This showed marked improvement in the total recordable injury frequency rate by the two zone contractors in 2014–15, when compared to not only Sydney region, but also to Hunter and Southern regions on RMS performed road maintenance in 2012–13.

Exhibit 6: Total Recordable Injury Frequency Rate score comparison
Measure South zone West zone Sydney region Hunter region Southern region
2014-15 2012-13
TRIFR 19.6 5.11 56.59 45.31 57.25

Source: Roads and Maritime Services 2015.

During the course of the audit, RMS advised that it intends to repeat this type of study to determine whether it has achieved all expected benefits (and their value), and that it would use the results to inform its negotiation with the SMC contractors as part of the commercial reset due in mid-2018.