What this report is about 

This report assesses how effectively SafeWork NSW, a part of the Department of Customer Service (DCS), has performed its regulatory compliance functions for work health and safety in New South Wales. 

The report includes a case study examining SafeWork NSW's management of a project to develop a realtime monitoring device for airborne silica in workplaces. 

Findings 

There is limited transparency about SafeWork NSW's effectiveness as a regulator. The limited performance information that is available is either subsumed within DCS reporting (or other sources) and is focused on activity, not outcomes. 

As a work health and safety (WHS) regulator, SafeWork NSW lacks an effective strategic and data-driven approach to respond to emerging WHS risks. 

It was slow to respond to the risk of respirable crystalline silica in manufactured stone. 

SafeWork NSW is constrained by an information management system that is over 20 years old and has passed its effective useful life. 

While it has invested effort into ensuring consistent regulatory decisions, SafeWork NSW needs to maintain a focus on this objective, including by ensuring that there is a comprehensive approach to quality assurance. 

SafeWork NSW's engagement of a commercial partner to develop a real-time silica monitoring device did not comply with key procurement obligations. 

There was ineffective governance and process to address important concerns about the accuracy of the real-time silica monitoring device. 

As such, SafeWork NSW did not adequately manage potential WHS risks. 

Recommendations 

The report recommended that DCS should: 

• ensure there is an independent investigation into the procurement of the research partner for the real-time silica detector 
• embed a formal process to review and set its annual regulatory priorities 
• publish a consolidated performance report 
• set long-term priorities, including for workforce planning and technology uplift 
• improve its use of data, and start work to replace its existing complaints handling system 
• review its risk culture and its risk management framework 
• review the quality assurance measures that support consistent regulatory decisions

Read the PDF report.

Parliamentary reference - Report number #390 - released 27 February 2024
 

What this report is about

Extreme rainfall across eastern Australia in 2021 and 2022 led to a series of major flood events in New South Wales.

This audit assessed how effectively the NSW Government provided emergency accommodation and temporary housing in response to the early 2022 Northern Rivers and late 2022 Central West flood events.

Responsible agencies included in this audit were the Department of Communities and Justice, NSW Reconstruction Authority, the former Department of Planning and Environment, the Department of Regional NSW and the Premier’s Department.

Findings

The Department of Communities and Justice rapidly provided emergency accommodation to displaced persons immediately following these flood events.

There was no plan in place to guide a temporary housing response and agencies did not have agency-level plans for implementing their responsibilities.

The NSW Government rapidly procured and constructed temporary housing villages. However, the amount of temporary housing provided did not meet the demand.

There is an extensive waitlist for temporary housing and the remaining demand in the Northern Rivers is unlikely to be met. The NSW Reconstruction Authority has not reviewed this list to confirm its accuracy.

Demobilisation plans for the temporary housing villages have been developed, but there are no long-term plans in place for the transition of tenants out of the temporary housing.

Agencies are in the process of evaluating the provision of emergency accommodation and temporary housing.

The findings from the 2022 State-wide lessons process largely relate to response activities.

Audit recommendations

The NSW Reconstruction Authority should:

• Develop a plan for the provision of temporary housing.
• Review the temporary housing waitlist.
• Determine a timeline for demobilising the temporary housing villages.
• Develop a strategy to manage the transition of people into long-term accommodation.
• Develop a process for state-wide recovery lessons learned.

All audited agencies should:

• Finalise evaluations of their role in the provision of emergency accommodation and temporary housing.
• Develop internal plans for implementing their roles under state-wide plans.

Read the PDF report

Parliamentary reference - Report number #389 - released 22 February 2024

What this report is about

The report assesses whether the Department of Customer Service (the department) complied with legislation and NSW government policy when it directly negotiated with Duncan Solutions to procure backend services relating to the Park'nPay app.

The Park'nPay app, developed by the department, enables users to locate and pay for parking remotely using their smart mobile device.

The audit found

The department failed to establish the grounds for entering a direct negotiation procurement strategy, without any competitive tendering, for services for the Park'nPay app. It rushed a decision to trial the app in The Rocks, without considering how this might affect its procurement obligations.

There is no evidence that the procurement achieved value for money. Despite being required by legislation, as well as mandatory NSW government policy, the department did not consider how it would ensure value for money, nor did it demonstrate an adequate understanding of what is meant by value for money on this occasion.

The department failed to implement key probity requirements. There was no effective management of conflicts of interest. Key decisions were not documented. There was a lack of clarity, transparency, and oversight of the relationship between the Minister's office and staff in the department.

The audit made recommendations about

1. making and retaining complete and accurate records, particularly on decisions to commit or expend public money
2. ensuring department staff understand how to exercise their financial delegations and procurement processes
3. ensuring that only staff with appropriate delegations are committing or approving the spending of public money
4. consistency with the contract extension provisions of the NSW Government Procurement Policy Framework, particularly regarding ensuring value for money
5. protocols to guide the interactions between department staff and Minister and Minister's staff
6. the need for proper management and oversight of contingent workers, such as contractors.

On 27 February 2019 the then Minister for Finance, Services and Property announced the commencement of a Park’nPay app trial in The Rocks precinct of Sydney.

The app was intended to enable users to locate and pay for parking remotely, using their smart mobile device such as a phone or tablet, rather than needing to physically be at a parking meter.

In July 2019, following a direct negotiation procurement conducted by the then Department of Finance, Services and Innovation, a contract was executed with Duncan Solutions for an estimated value of $1,260,600 over three-years, with three single-year options to extend. The contract required Duncan Solutions to provide development services to link the Park'nPay app to its Parking Enterprise Management System platform and to provide ongoing software support services.

This audit assessed whether the department complied with the procurement obligations that applied at the time it procured these services from Duncan Solutions.

This audit focussed on the department's processes and decision-making relating to:

• the direct negotiation with Duncan Solutions at the exclusion of any other potential supplier
• the negotiation, execution and management of the contract with Duncan Solutions.

As this audit focusses on the department's procurement and contract management processes, it does not comment on the activities of Duncan Solutions. The detailed audit objective, criteria and audit approach are in Appendix three.

The auditee is the Department of Customer Service. As a result of machinery of government changes, the Department of Finance, Services, and Innovation became the Department of Customer Service from 1 July 2019. To avoid confusion, this report simply uses ‘the department’ to refer to either. Where the report refers to the Minister, it relates to the former Minister in office at the time.

Appendix one – Response from auditee

Appendix two – Key requirements of the department's procurement manual 

Appendix three – About the audit 

Appendix four– Performance auditing

Copyright notice

© Copyright reserved by the Audit Office of New South Wales. All rights reserved. No part of this publication may be reproduced without prior consent of the Audit Office of New South Wales. The Audit Office does not accept responsibility for loss or damage suffered by any person acting on or refraining from action as a result of any of this material.

Parliamentary reference - Report number #387 - released 14 December 2023

What the report is about

This audit assessed how effectively NSW government agencies procure and manage consultants. It examined the role of the NSW Procurement Board and NSW Procurement (a unit within NSW Treasury) in supporting and monitoring agency procurement and management of consultants.

The audit used four sources of data that contain information about spending on consultants by NSW government agencies, including annual report disclosures and the State's financial consolidation system (Prime). It also reviewed a sample of consulting engagements from ten NSW government agencies.

What we found

Our review of a selection of consulting engagements indicates that agencies do not procure and manage consultants effectively.

We found most agencies do not use consultants strategically and do not have systems for managing or evaluating consultant performance. We also found examples of non-compliance with procurement rules, including contract variations that exceeded procurement thresholds.

NSW Procurement has made improvements to the information available about spending on consultants, including additional analysis and reporting. However, there is no single data source that accurately captures spending on consultants.

Our analysis of data on whole-of-government spending on consultants, drawn from agency annual reports, indicates that four large professional services firms accounted for about a quarter of consultancy expenditure from 2017–18 to 2021–22. This concentration increases strategic risks, including over-reliance on a limited number of providers and potential reduction in the independence of advice.

It is also highly unlikely that NSW government agencies will meet the government's 2019 policy commitment to reduce consultancy expenses by 20% each year, over four years, from 2019–20. NSW Treasury advised that to implement this commitment, agency budgets were reduced in Prime in line with the savings targets. However, actual spending on consulting in NSW Treasury's Reports on State Finances 2020–21 and 2021–22 was almost $100 million higher than the savings targets over the first three years since 2019–20.

What we recommended

The report made seven recommendations which aim to improve:

• the quality and transparency of data on spending on consultants
• monitoring of strategic risks and agency compliance with procurement and recordkeeping rules
• agencies' strategic use of consultants, including evaluation and knowledge retention.

Between 2017–18 and 2021–22, NSW government agency annual reports disclosed total spending of around $1 billion on consultants across more than 10,000 engagements. More than 1,000 consulting firms provided services to NSW government agencies during this period. Consulting is a classification of professional services that is characterised by giving advice or recommendations on a specific issue. The NSW Procurement Board Direction PBD-2021-03 defines a consultant as a person or organisation that provides 'recommendations or professional advice to assist decision-making by management'. PBD-2021-03 notes that the advisory nature of the work of consultants is the main factor that distinguishes them from other providers of professional services.

The NSW Procurement Board is responsible for setting procurement policy, issuing directions to support policies, and monitoring and reporting on agency compliance with policies and directions. NSW Procurement, a division within NSW Treasury, supports agencies to comply with the NSW Procurement Board’s policies and directions. A 'devolved governance model' is used for procurement in New South Wales. This means the heads of government entities that are covered by the NSW Procurement Board’s directions are responsible for managing the entity's procurement, including managing risks, reporting and ensuring compliance, in line with procurement laws and policies.

This audit assessed how effectively NSW government agencies procure and manage consultants. It assessed the role of the NSW Procurement Board and NSW Procurement in supporting and monitoring agency procurement and management of consultants. It also reviewed a sample of consulting engagements from ten NSW government agencies to examine how agencies procured, managed and reported on their use of consultants. The ten NSW government agencies were:

• NSW Treasury
• Department of Communities and Justice
• Department of Customer Service
• Department of Education
• Department of Planning and Environment
• Department of Premier and Cabinet
• Department of Regional NSW
• Infrastructure NSW
• Sydney Metro
• Transport for NSW

There are four different sources of data that contain information about spending on consultants by NSW government agencies: the State's financial consolidation system (Prime), disclosures of spending on consultants in agency annual reports, and two systems operated by NSW Procurement (the Business Advisory Services (BAS) dashboard and Spend Cube). Each of these data sources serves a different purpose, and collects and categorises information differently. None of these provide a complete source of data on spending on consultants, either in their own right or collectively.

NSW Treasury considers Prime to be the 'source of truth' on consulting expenditure across the NSW public sector. An account within Prime records recurrent spending on consultants, but this account does not include capital expenditure (that is, spending on consultants that has from a financial reporting perspective been 'capitalised' to a project on the balance sheet). As the State's financial consolidation system, Prime captures all financial information. However, capitalised consulting expenditure is recorded within various capital accounts, and is not identifiable within these accounts. While this is appropriate for accounting purposes, it means that the Prime account that records recurrent consulting expenditure does not reflect total spending on consultants by NSW government agencies. We used the data in Prime to assess whether NSW government agencies met the NSW Government's policy commitment—stated before the 2019 election and costed by the Parliamentary Budget Office—to reduce recurrent expenditure on consulting by 20% each year, over four years, from 2019–20. We did this because, while the Prime account for recurrent consulting expenditure does not reflect all spending on consultants, it does capture the recurrent spending that was subject to the policy commitment.

Most NSW government agencies are required by legislation to disclose spending on consultants (as defined in PBD-2021-03) in their annual reports. These disclosures include both recurrent and capital expenditure. For consulting engagements that cost more than $50,000, the disclosures also provide itemised information, including the names of the individual projects and the consultants used. While this data is more complete than Prime because it includes capital expenditure, it also has some gaps. Some entities are excluded from public reporting requirements on consultant use. For example, NSW Local Health Districts (LHD) are not required to produce annual reports, and the Ministry of Health does not include LHD consulting expenditure in its annual report.¹ We used annual report disclosure data to report on total expenditure on consultants, and the concentration of suppliers of consulting services to NSW government agencies.

The BAS dashboard and Spend Cube are systems created by NSW Procurement to collect information about spending on suppliers of professional services. This includes consultants, but also includes other professional services providers. The systems were not designed for reporting on spending on consulting as defined in PBD-2021-03. However, we have used this data to assess specific aspects of NSW Procurement's monitoring of the use of consultants by NSW government agencies.

In 2018, we conducted an audit titled 'Procurement and reporting of consultancy services'. This assessed how 12 NSW government agencies complied with procurement requirements and how NSW Procurement supported the functions of the NSW Procurement Board. The 2018 audit found that none of the 12 agencies fully complied with NSW Procurement Board Directions on the use of consultants and that the NSW Procurement Board was not fully effective in overseeing and supporting agencies’ procurement of consultants. Specific findings from the 2018 audit included: 

• Agencies applied the definition of consultant inconsistently, which affected the accuracy of reporting on consultancy expenditure.
• There was inadequate guidance from NSW Procurement for agencies implementing the procurement framework, with a need for additional tools, automated processes, and other internal controls to improve compliance.
• NSW Procurement had insufficient data for effective oversight of procurement and did not publish any data on the procurement of consultancy services by NSW government agencies.

This chapter outlines our findings on the role of NSW Procurement in overseeing the use of consultants by NSW government agencies.

This chapter outlines our findings on the use of consultants by the ten NSW government agencies that were included in this audit.

Appendix one – Responses from auditees

Appendix two – About the audit

Appendix three – Performance auditing

Copyright notice

© Copyright reserved by the Audit Office of New South Wales. All rights reserved. No part of this publication may be reproduced without prior consent of the Audit Office of New South Wales. The Audit Office does not accept responsibility for loss or damage suffered by any person acting on or refraining from action as a result of any of this material.

Parliamentary reference - Report number #378 - released 2 March 2023

What the report is about

The Government Advertising Act 2011 requires the Auditor-General to undertake a performance audit on government advertising activities each financial year.

This audit examined whether TAFE NSW's annual advertising campaign in 2021–22:

1. was carried out effectively, economically, and efficiently
2. complied with regulatory requirements and the Government Advertising Guidelines.

What we found

TAFE NSW complied with Section 6 of the Act, prohibiting political content.

It also complied with most other advertising requirements.
 
An important exception was that the Managing Director certified that the campaign complied with regulatory requirements and was an efficient and cost-effective means of achieving its public purpose, before a cost-benefit analysis (CBA) was completed.

We have found issues with agencies complying with CBA requirements in previous government advertising audits. This includes the failure to complete them before signing compliance certificates.

The policy owner, the Department of Customer Service (DCS), does not consider oversight of CBAs to be within the scope of their peer review process.  

TAFE NSW evaluated this advertising campaign by surveying a population significantly broader than the target audience. As such, survey results may not accurately reflect the views of the intended audience.

What we recommended

By 30 June 2023, TAFE NSW should:

1. implement processes that ensure:
   1. CBAs are completed before the launch of campaigns over $1 million
   2. compliance certificates are completed only after all regulatory requirements are met
2. consider adding to its current evaluation methods by surveying a population which closely reflects the age profile of its intended target audience.

By June 2023, DCS should:

3. improve whole‑of‑government reporting and monitoring processes to provide the NSW Government with a central view of compliance, including the completion of CBAs by agencies.

The Government Advertising Act 2011 (the Act) sets out requirements that must be followed by a government agency when it carries out a government advertising campaign. The requirements include an explicit prohibition on political advertising, as well as a need to complete a peer review and cost-benefit analysis before the campaign commences. The accompanying Government Advertising Regulation 2018 (the Regulation) and Government Advertising Guidelines (the Guidelines) address further matters of detail.

The Act also requires the Auditor-General to conduct a performance audit on the activities of one or more government agencies in relation to government advertising campaigns in each financial year. The performance audit must assess whether a government agency (or agencies) has carried out activities in relation to government advertising campaigns in an effective, economical and efficient manner. It also assesses compliance with the Act, the Regulation, other laws and the Guidelines.

This audit examined TAFE NSW's advertising campaign for the 2021–22 financial year. TAFE NSW is the NSW Government's public provider of vocational education and training. TAFE NSW carries out an advertising campaign every year. In 2021–22, it spent $15.16 million on developing and implementing advertising. TAFE NSW used channels such as television, radio, internet and social media, press, and out of home advertising in public settings such as bus stops. The advertising aimed to increase the percentage of people considering TAFE NSW for training or education, grow the percentage of people who consider TAFE NSW to be the preferred education provider in NSW, and maintain the proportion of people who are aware of TAFE NSW more generally.

There are a range of private service providers helping to deliver vocational education and training in NSW.

This part of the report sets out key aspects of TAFE NSW's compliance with the government advertising regulatory framework. It considers whether TAFE NSW complied with the:

• Government Advertising Act 2011
• Government Advertising Regulation 2018
• NSW Government Advertising Guidelines 2012 and other relevant policy.

This part of the report considers whether TAFE NSW's advertising program for 2021–22 was carried out in an effective, efficient, and economical manner.

Appendix one – Responses from agencies

Appendix two – About the campaign

Appendix three – About the audit

Appendix four – Performance auditing

Copyright notice

© Copyright reserved by the Audit Office of New South Wales. All rights reserved. No part of this publication may be reproduced without prior consent of the Audit Office of New South Wales. The Audit Office does not accept responsibility for loss or damage suffered by any person acting on or refraining from action as a result of any of this material.

Parliamentary reference - Report number #377 - released 28 February 2023

What the report is about

The Transport Asset Holding Entity (TAHE) is the State's custodian of rail assets. It is a state owned corporation and commenced operating on 1 July 2020.

This audit assessed the effectiveness of NSW Government agencies' design and implementation of TAHE. We audited TAHE, Transport for NSW (TfNSW) and NSW Treasury.

Separate and related audits on TAHE are reported in 'State Finances 2022', 'State Finances 2021' and 'Transport and Infrastructure 2022' reports.

What we found

The design and implementation of TAHE, which spanned seven years, was not effective.

The process was not cohesive or transparent. It delivered an outcome that is unnecessarily complex in order to support an accounting treatment to meet the NSW Government's short-term Budget objectives, while creating an obligation for future governments.

The benefits of TAHE were claimed in the 2015–16 NSW Budget before the enabling legislation was passed by Parliament in 2017. This committed the agencies to implement a solution that justified the 2015–16 Budget impacts, regardless of any challenges that arose.

Rail safety arrangements were a priority throughout TAHE's design and implementation, and risks were raised and addressed.

Agencies relied heavily on consultants on matters related to the creation of TAHE, but failed to effectively manage these engagements. Agencies failed to ensure that consultancies delivered independent advice as an input to decision-making. A small number of firms were used repeatedly to provide advice on the same topic. The final cost of TAHE-related consultancies was $22.6 million compared to the initial estimated cost of $12.9 million.

What we recommended

We recommended that the audited agencies should:

• improve accountability and transparency for major new fiscal transformation initiatives
• ensure entities do not reflect the financial impact of significant initiatives in the Budget when there is uncertainty, or it creates perverse incentives
• review record keeping practices, systems and policies to ensure compliance with the State Records Act 1998, and the NSW Government Information Classification, Labelling and Handling Guidelines
• review procurement policies to ensure that consultant use complies with all NSW Government policy requirements.

The NSW Government established the Transport Asset Holding Entity (TAHE), a statutory State Owned Corporation (SOC), on 1 July 2020 to replace the former rail infrastructure owner – RailCorp. It is the State's custodian of rail network assets, including rail tracks and other infrastructure, rolling stock, land, train stations and facilities, retail space, and signal and power systems, within metropolitan and regional New South Wales. It is responsible for $2.8 billion of major capital projects in 2022–23.

TAHE was established under Part 2 of the Transport Administration Act 1988 and is governed by a decision-making board. The Treasurer and the Minister for Finance and Employee Relations are the Shareholding Ministers of TAHE, and they annually agree performance expectations articulated in a Statement of Corporate Intent.

Whereas TAHE is the custodian of rail assets, Sydney Trains and NSW Trains operate public rail services. TAHE does not have responsibility for the operation of the heavy rail network or train services, nor does it have network control functions. TAHE, Sydney Trains and NSW Trains are in the Transport and Infrastructure cluster in the public sector (formerly the Transport cluster and renamed in April 2022), which also includes Sydney Metro and Transport for NSW (TfNSW).

TfNSW leads the Transport and Infrastructure cluster. Its role is to set the strategic direction for transport across the State. This involves the shaping of planning, policy, strategy, regulation, resource allocation and other service and non-service delivery functions for all modes of transport.

TAHE's Operating Licence is granted by the Portfolio Minister and authorises the entity to perform the functions required to acquire, develop, finance, divest and hold assets, pursuant to the Transport Administration Act 1988. The Portfolio Minister also issues a Statement of Expectations which outlines the government’s expectation for the business for the next three to five years.

TAHE's original Portfolio Minister was the Minister for Transport who approved, on 30 June 2020, the issuing of an interim 12-month Operating Licence to enable TAHE to commence operating on 1 July 2020. The Portfolio Minister then granted TAHE's current Operating Licence in 2021. After TAHE requested a 12-month extension to its current Operating Licence, its next Operating Licence is due on 1 July 2024. The current Portfolio Minister is the Minister for Infrastructure, Cities and Active Transport.

About this audit

This audit assessed the effectiveness of NSW Government agencies' design and implementation of TAHE. In making this assessment, we considered whether: 

• the process of designing and implementing TAHE was cohesive and transparent, and delivered an effective outcome
• agencies' roles and responsibilities were clear in the planning of TAHE
• agencies effectively identified and managed certain risks.

Appendix one – Responses from audited agencies, and Audit Office clarification of matters raised in the TAHE formal response 

Appendix two – Classification of government entities 

Appendix three – About the audit 

Appendix four – Performance auditing

Copyright notice

© Copyright reserved by the Audit Office of New South Wales. All rights reserved. No part of this publication may be reproduced without prior consent of the Audit Office of New South Wales. The Audit Office does not accept responsibility for loss or damage suffered by any person acting on or refraining from action as a result of any of this material.

Parliamentary reference - Report number #372 - released 24 January 2023

What the report is about

This audit assessed how effectively Transport for NSW (TfNSW) and Sydney Trains identify and manage their cyber security risks.

The NSW Cyber Security Policy (CSP) sets out 25 mandatory requirements for agencies, including implementing the Australian Cyber Security Centre’s Essential 8 strategies to mitigate cyber security incidents, and identifying the agency’s most vital systems, their ‘crown jewels’. 

The audited agencies have requested that we do not disclose detail of the significant vulnerabilities detected during the audit, as these vulnerabilities are not yet remediated. We provided a detailed report to the agencies in December 2020 outlining significant issues identified in the audit. We have conceded to the agencies' request but it is disappointing that transparency to the Parliament and the public on issues that potentially directly affect them needs to be limited in this way.

What we found

TfNSW and Sydney Trains are not effectively managing their cyber security risks.

Both agencies have assessed their cyber security risks as unacceptably high and both agencies had not identified all of the risks we detected during this audit – some of which are significant.

Both agencies have cyber security plans in place that aim to address cyber security risks. TfNSW and Sydney Trains have combined this into the Transport Cyber Defence Rolling Program, part of the Cyber Defence Portfolio (CDP). 

However, neither agency has reached its target ratings for the CSP and the Essential 8 and maturity is low in relation to significant risks and vulnerabilities exposed.

Further, neither agency is fostering a culture where cyber security risk management is an important and valued aspect of decision-making.

TfNSW is not implementing cyber security training effectively across the cluster with only 7.2% of staff having completed basic cyber security training.

What we recommended

TfNSW and Sydney Trains should:

• develop and implement a plan to uplift the Essential 8 controls to the agency's target state
• as a matter of priority, address the vulnerabilities identified as part of this audit and previously described in a detailed Audit Office report provided to both agencies
• ensure cyber security risk reporting to executives and the Audit and Risk Committee
• collect supporting information for the CSP self assessments 
• classify all information and systems according to importance and integrate this with the crown jewels identification process
• require more rigorous analysis to re-prioritise CDP funding 
• increase uptake of cyber security training.

TfNSW should assess the appropriateness of its target rating for each of the CSP mandatory requirements.

Department of Customer Service should:

• clarify the requirement for the CSP reporting to apply to all systems
• require agencies to report the target level of maturity for each mandatory requirement.

Response to requests by audited agencies to remove information from this report

In preparing this audit report, I have considered how best to balance the need to support public accountability and transparency with the need to avoid revealing information that could pose additional risk to agencies’ systems. This has involved an assessment of the appropriate level of detail to include in the report about the cyber security vulnerabilities identified in this audit.

In making this assessment, the audit team consulted with Transport for NSW (TfNSW), Sydney Trains, and Cyber Security NSW to identify content which could potentially pose a threat to the agencies’ cyber security.

In December 2020, my office also provided TfNSW and Sydney Trains with a detailed report of many of the significant vulnerabilities identified in this audit, to enable the agencies to address the cyber security risks identified. The detailed report was produced as a result of a 'red team' exercise, which was conducted with both agencies' knowledge and consent. The scope of this exercise reflected the significant input provided by both agencies. More information on this exercise is at page 12 of this report.

TfNSW and Sydney Trains have advised that in the six months from December 2020 and at the time of tabling this audit report, they have not yet remediated all the vulnerabilities identified. As a result, they, along with Cyber Security NSW, have requested that we not disclose all information contained in this audit report to reduce the likelihood of an attack on their systems and resulting harm to the community. I have conceded to this request because the vulnerabilities identified have not yet been remediated and leave the agencies exposed to significant risk.

It should be stressed that the risks identified in the detailed report exist due to the continued presence of these previously identified vulnerabilities, rather than due to their potential publication. The audited agencies, alone, are accountable for remediating these vulnerabilities and addressing the risks they pose.

It is disappointing that transparency to the Parliament and the public on issues that potentially directly affect them needs to be limited in this way.

That said, the conclusions drawn in this report are significant in terms of risk and remain valid, and the recommendations should be acted upon with urgency.

Cyber security risk is an increasing area of concern for governments in Australia and around the world. In recent years, there have been a number of high-profile cyber security attacks on government entities in Australia, including in New South Wales. Malicious cyber activity in Australia is increasing in frequency, scale, and sophistication. The Audit Office of New South Wales is responding to these risks with a program of audits in this area, which aim to identify the effectiveness of particular agencies in managing cyber risks, as well as their compliance with relevant policy.

Cyber Security NSW, part of the Department of Customer Service (DCS) releases and manages the NSW Cyber Security Policy (CSP). The CSP sets out 25 mandatory requirements for agencies, including making it mandatory for agencies to implement the Australian Cyber Security Centre Essential 8 Strategies to Mitigate Cyber Security Incidents (the Essential 8). The Essential 8 are key controls which serve as a baseline set of protections which agencies can put in place to make it more difficult for adversaries to compromise a system. Agencies are required to self-assess their maturity against the CSP and the Essential 8, and report that assessment to Cyber Security NSW annually.

The CSP makes agencies responsible for identifying and managing their cyber security risks. The CSP sets out responsibilities and governance regarding risk identification, including making agencies responsible for identifying their 'crown jewels', the agency's most valuable and operationally vital systems. Once these risks are identified, agencies are responsible for developing a cyber security plan to mitigate those risks.

This audit focussed on two agencies: Transport for NSW (TfNSW) and Sydney Trains. TfNSW is the lead agency for the Transport cluster and provides a number of IT services to the entire cluster, including Sydney Trains. This audit focussed on the activities of TfNSW's Transport IT function, which is responsible for providing cyber security across the cluster, as well as directly overseeing four of TfNSW's crown jewels. Sydney Trains is one of the agencies in the Transport cluster. While it receives some services from TfNSW, it is also responsible for implementing its own IT controls, as well as controls to protect its Operational Technology (OT) environment. This OT environment includes systems which are necessary for the operation and safety of the train network.

To test the mitigations in place and the effectiveness of controls, this audit involved a 'red team' simulated exercise. A red team involves authorised attackers seeking to achieve certain objectives within the target's environment. The red team simulated a determined external cyber threat actor seeking to gain access to TfNSW's systems. The red team also sought to test the physical security of some Sydney Trains' sites relevant to the agency's cyber security. The red team exercise was conducted with the knowledge of TfNSW and Sydney Trains.

This audit included the Department of Customer Service as an auditee, as they have ownership of the CSP through Cyber Security NSW. This audit did not examine the management of cyber risk in the Department of Customer Service.

This audit assessed how effectively selected agencies identify and manage their cyber security risks. The audit assessed this with the following criteria:

• Are agencies effectively identifying and planning for their cyber security risks?
• Are agencies effectively managing their cyber security risks?

Following this in-depth portfolio assessment, the Auditor-General for NSW will also table a report on NSW agencies' compliance with the CSP in the first quarter of 2021–22.

Agencies have assessed their cyber risks as being above acceptable levels

An agency's risk tolerance is the amount of risk which the agency will accept or tolerate without developing further strategies to modify the level of risk. Risks that are within an agency's risk tolerance may not require further mitigation and may be deemed acceptable, while risks which are above the agency's risk tolerance likely require further mitigation before they become acceptable to the agency.

Both agencies have defined their risk tolerance and have identified risks which are above this level, indicating that they are unacceptable to the agency. TfNSW has defined 'very high' risks as generally intolerable and 'high' risks as undesirable. Its risk tolerance is 'medium'. Sydney Trains has four classifications of risk: A, B, C and D. A and B risks are deemed 'unacceptable' and 'undesirable' respectively, while C risks are considered 'tolerable'. This aligns with the TfNSW definition of a medium risk tolerance.

Transport IT reported five enterprise-level cyber security risks through its enterprise risk reporting tool in September 2020, all of which relate to cyber security or have causes relating to cyber security. These risks are in aggregate form, rather than relating to specific vulnerabilities. At the time of the audit, one of these risks was rated as very high and the other four rated as high. At this time, Transport IT had identified a further seven divisional-level risks which were above the agency’s risk tolerance.

Similarly, Sydney Trains has identified one main cyber security risk in its IT enterprise-level risk register and another with a potential cyber cause. Both of these IT risks are deemed to have a residual risk of ‘unacceptable’.

Similarly, two cyber-related OT risks have been determined to be above the agency's risk tolerance. One risk is rated as 'unacceptable'. Another risk, while not entirely cyber rated, is rated 'undesirable' and is deemed to have some causes which may stem from a cyber-attack.

Agencies have assessed their current cyber risk mitigations as requiring improvement

In addition to the risk ratings stated above, at the time of the audit neither agency believed that its controls were operating effectively. Transport IT had rated the control environments for its cyber security enterprise risks as 'requires improvement'. Mitigations were listed in the risk register for these risks but, in some cases, they were unlikely to reduce the risk to the target state or by the target date. For example, one risk had actions listed as 'under review' and no further treatment actions listed, but a due date of July 2021, while another risk was being treated by the CDP with a due date of July 2021. The CDP identified in May 2020 that while the average risk identified as part of that program will be reduced to a medium level by this date, ten high risks will still remain. Given the delays in the program, this number may be higher. As such, it seems unlikely that the enterprise risk will be reduced to below a 'high' level by July 2021.

Sydney Trains’ IT and OT risk registers cross-reference controls and mitigations against the causes and consequences. The IT cyber security risk identified in the register had causes with no mitigations designed for them. Further, some of these causes did not have future mitigations designed for them. This risk also had controls in place which are identified as partially effective. For the unacceptable OT risk noted above, while there was a control designed for each of the potential causes, Sydney Trains had identified all of the controls in place as either partially effective or ineffective. This indicates that Sydney Trains was not effectively mitigating the causes of its cyber risks and, even where it had designed controls or mitigations, these were not always implemented to fully mitigate the cause of the risk.

Additional information on gaps in cyber mitigations which were exposed in the course of this audit has been detailed to both agencies. The Foreword of this report provides information about why this detail is not included here.

Essential 8 maturity is low across TfNSW and Sydney Trains and little progress was made in 2020

CSP mandatory requirement 3.2 states that agencies must implement the ACSC Essential 8. Agencies must also rate themselves against each of the Essential 8 on a maturity scale from zero to three and report this to Cyber Security NSW. A full list of the Essential 8 can be found in Exhibit 1. Both agencies have a low level of maturity against the Essential 8 not just in comparison to the targets they have set, but also in relation to the risks and vulnerabilities exposed. Both agencies have set target maturity ratings for the Essential 8 but none of the Essential 8 ratings across either agency are currently implemented to this level. Having a low level of Essential 8 maturity exposes both agencies to significant risks and vulnerabilities. Little progress was made between the 2019 and 2020 attestation periods.

Transport IT has set a target rating of three across all of the Essential 8. Sydney Trains has set a target rating of three for its IT systems. Sydney Trains had an interim target of two for its OT systems in 2020 and advised that this has since increased to three. It should be noted that not all the Essential 8 are applicable to OT systems.

None of the Essential 8 ratings across either agency are currently implemented to the target levels. Given that the Essential 8 provide the controls which are most commonly able to deter cyber-attacks, having maturity at a low level potentially exposes agencies to a cyber security attack.

Some work is underway across both TfNSW and Sydney Trains to improve the Essential 8 control ratings. The CDP provided some resources to the Essential 8 over 2019–20, with uplift focusing on specific systems. The CDP work in 2019 and 2020 relevant to the Essential 8 largely focussed on determining the current state of the Essential 8 and creating a target state roadmap. As a result, there was little improvement between the 2019 and 2020 attestation periods. The CDP has a workstream for the Essential 8 in its FY 2020–21 funding allocation, however as noted above in Exhibit 6 this was delayed as resources were redeployed to Project La Brea. Regardless, work on some specific aspects of the Essential 8 remain part of the 2020–21 CDP allocation, with workstreams allocated to improving three of the Essential 8. In addition, some work from Project La Brea should lead to an improvement in the Essential 8.

Sydney Trains' Cyber Uplift Program included a workstream which had in scope the uplift in the Essential 8 in IT. There were also other workstreams which aimed to improve some of the Essential 8 for OT systems. Work is also ongoing as part of the CDP to uplift these scores in Sydney Trains.

TfNSW and Sydney Trains have not reached their target maturity across the CSP mandatory requirements and TfNSW has not evaluated its cluster-wide target to ensure it is appropriate

Cyber Security NSW allows each agency to determine its target level of maturity for the first 20 CSP mandatory requirements. Agencies can tailor their target levels to their risk profile. Not reaching the target rating of the CSP mandatory requirements risks information and systems being managed inconsistently or not in alignment with good governance principles.

Sydney Trains has set its target level of maturity for IT and OT. All of Sydney Trains' target maturity levels are at least a three (defined), with a target of four (quantitatively managed) for many of the mandatory requirements. While Cyber Security NSW does not currently mandate a minimum level of maturity, in 2019 there was a requirement for each agency to target a minimum level of three.

Sydney Trains has not met its target ratings across the mandatory requirements.

The Transport Cyber Defence Rolling Program has a program KPI to ensure that the entire cluster reaches a minimum maturity level of three against all the CSP requirements by 2023. TfNSW has not reviewed its CSP mandatory requirement targets to determine if a three is desirable for all requirements or if a higher target level may be more appropriate. It is important for senior management to set cyber security objectives as a demonstration of leadership and a commitment to cyber security.

TfNSW has not met its target ratings across the mandatory requirements for its Group IT ISMS, which was the focus of this audit.

Both agencies claimed progress in their implementation of the mandatory requirements between 2019 and 2020. The audit did not seek to verify the self-assessed results from either agency.

Both agencies operate ISMS in line with the CSP

CSP mandatory requirement 3.1 requires agencies to implement an Information Security Management System (ISMS) or Cyber Security Framework (CSF), with scope at least covering systems identified as the agency's ‘crown jewels’. The ISMS or CSF should be compliant with, or modelled on, one or more recognised IT or OT standard. As noted in the introduction, an ISMS ‘consists of the policies, procedures, guidelines, and associated resources and activities, collectively managed by an organisation, in the pursuit of protecting its information assets.’ Both agencies operate an ISMS compliant with the CSP requirement.

As noted in the introduction, TfNSW operates four ISMS. The Transport IT ISMS is certified against ISO27001, the most common standard for ISMS certification. Three of TfNSW’s six crown jewels are managed within this ISMS. The other ISMS are not certified to relevant standards, though TfNSW claims that they align with relevant controls. This is sufficient for the purposes of the CSP.

Sydney Trains operates two ISMS, one for IT and another for OT. Neither of these are certified to relevant ISMS Standards, however there have been conformance reviews of both IT and OT with relevant standards. These ISMS cover all crown jewels in the agency.

There are currently 11 ISMS in operation across the Transport cluster. TfNSW has proposed moving towards a holistic approach to these ISMS, with the CDP Board responsible for governing the available security controls and directing agency IT and OT teams to implement these.

Agencies are not routinely conducting audits of third-party suppliers to ensure compliance with contractual obligations

CSP mandatory requirement 1.5 makes agencies accountable for the cyber risks of their ICT service providers and ensuring that providers comply with the CSP and any other relevant agency security policies. The ACSC has provided advice on what organisations should do when managing third party suppliers of ICT. The ACSC advises that organisations should use contracts to define cyber security expectations and seek assurance to ensure that these contract expectations are being met. While both agencies usually include specific cyber security expectations in contracts, neither is routinely seeking assurance that these expectations are being met.

The NSW Government has mandated the use of the 'Core& One' contract template for low-value IT procurements and the Procure IT contract template for high-value IT procurements. Both of these contracts contain space for the procuring agency to include cyber security controls for the contractor to implement. The Procure IT contract template also includes a right-to-audit clause which allows agencies to receive assurance around the implementation of these controls. TfNSW and Sydney Trains used the mandated contracts for relevant contracts examined as part of this audit.

TfNSW included security controls in all the contracts examined as part of this audit. Compliance with ISO27001 was the most commonly stated security expectation. Of the contracts examined as part of this audit, only one contract did not have a right-to-audit clause. This contract was signed in October 2016. While these clauses are in place, TfNSW rarely conducted these audits on its third-party providers. Of the eight TfNSW contracts examined in detail, only two of these had been audited to confirm compliance with the stated security controls.

Sydney Trains included security controls in all but one of the contracts examined as part of this audit. Sydney Trains did not require contractors to be compliant with ISO27001, but only required compliance with whole-of-government policies. Sydney Trains does not routinely conduct audits of its third-party suppliers, however it did conduct deep-dive risk analyses of its top ten highest risk IT suppliers. This involved a detailed review of both the suppliers' security posture and also the contract underpinning the relationship with the supplier.

The CDP funding for 2020–21 includes a workstream for strategic third-party contract remediation. This funding is to conduct some foundational work which will allow the CDP to make further improvements in future years. While this funding will not address gaps in contract requirements or management across all contracts, this workstream aims to reduce the risks posed by strategic suppliers covering critical assets. Similarly, work is currently underway as part of the CDP to conduct OT risk assessments for key suppliers to Sydney Trains in a similar way to the work undertaken for IT suppliers.

Sydney Trains has risk assessed its third-party suppliers but TfNSW has not done so

It is important to conduct a risk assessment of suppliers to identify high-risk contractors. This allows agencies to identify those contractors who may require additional controls stated in the contract, those who require additional oversight, and also where auditing resources are best targeted.

Sydney Trains has risk assessed all its IT suppliers and, as noted above, has conducted a deep-dive risk analysis of its top ten highest risk suppliers. TfNSW has not undertaken similar analysis of its key suppliers, however it has identified risks attached to each of its strategic suppliers and has documented these. As a result of not risk assessing its suppliers, TfNSW cannot take a targeted approach to its contract management.

TfNSW demonstrated poor records handling relating to the contracts examined as part of this audit

TfNSW was not able to locate one of the contracts requested as part of the audit's sample. Other documentation, such as contract management plans, could not be located for many of the other contracts requested as part of this audit. These poor document handling practices limits TfNSW's ability to effectively oversee service providers and ensure that they are implementing agreed controls. It also limits public transparency on the effectiveness of these controls.

The Transport cluster is not effectively implementing cyber security awareness training

Agencies are responsible for implementing regular cyber security education for all employees and contractors under mandatory requirement 2.1 in the CSP. TfNSW is responsible for delivering this training to the whole Transport cluster, including Sydney Trains. The Transport cluster has basic cyber awareness training available for all staff. TfNSW also offers additional training provided by Cyber Security NSW targeted at executives and executive assistants. While TfNSW has training available to staff, it is not delivering this effectively. TfNSW does not make training mandatory for most staff nor does it require staff to repeat training regularly. Even among those staff who have been assigned the training, completion rates are low, meaning that delivery is not effectively monitored. Cyber security training is important for building and supporting a cyber security culture.

TfNSW is responsible for creating and rolling out all forms of training to agencies within the Transport cluster. Both TfNSW and Sydney Trains have the same mandatory cyber awareness training that is automatically assigned to new starters. At the time of the audit, this training was not mandatory for ongoing staff. TfNSW does make additional cyber security training available to staff who can choose to undertake the training themselves, or can be assigned the training by their manager. All TfNSW cyber security training is delivered via online modules and it is the responsibility of managers to ensure that it is completed.

Cyber security training completion rates for both TfNSW and Sydney Trains are low. Only 13.5 per cent of staff across the Transport cluster had been assigned the Cyber Safety for New Starters training as of January 2021. Although this course is mandatory for new starters, only 53 per cent of staff assigned the Cyber Safety for New Starters training module had completed the course by January 2021. As a result, only 7.2 per cent of staff across the entire Transport cluster had completed this training at that time. In Sydney Trains, less than one per cent of staff had completed this training as at January 2021 and a further 7.6 per cent of staff have completed the 'Cyber Security: Beyond the Basics' training. These low completion rates indicate that TfNSW is not effectively rolling out cyber security training across the cluster.

In October 2020, the Department of Customer Service released 'DCS-2020-05 Cyber Security NSW Directive - Practice Requirement for NSW Government', which made annual cyber security training mandatory for all staff from 2021. In line with this requirement, TfNSW has advised that it will be gradually implementing mandatory annual training from July 2021 for all staff.

The Transport cluster undertakes activities to build a cyber-aware culture in accordance with the CSP, but awareness remains low

Increasing staff awareness of cyber security risks and maintaining a cyber secure culture are both mandatory requirements of the CSP. While TfNSW does undertake some activities to build a cyber aware culture, awareness of cyber security risks remains low. This can be demonstrated by the low training rates outlined above, and the 'Spot the Scammer' exercise, described in Exhibit 7. TfNSW is responsible for delivering these awareness raising activities across the cluster.

TfNSW frequently communicates with staff across the Transport cluster about various cyber security risks through multiple avenues. Both agencies use the intranet, emails and other awareness raising activities to highlight the importance for staff to be aware of the seriousness of cyber risks. Advice given on the intranet includes tips for spotting scammers on mobile phones, promoting the cluster-wide training courses, as well as various advice that staff could use when dealing with cyber risks in the workplace.

In addition to these awareness raising activities, TfNSW has also undertaken a cluster-wide phishing email exercise called 'Spot the Scammer'. This is outlined in Exhibit 7. This exercise was carried out in 2019 and 2020 and allowed the Transport cluster to measure the degree to which staff were able to identify phishing emails. As can be seen in Exhibit 7, the results of this exercise indicate that staff awareness of phishing emails remains low.

Appendix one – Response from agencies

Appendix two – Cyber Security Policy mandatory requirements

Appendix three – About the audit

Appendix four – Performance auditing

© Copyright reserved by the Audit Office of New South Wales. All rights reserved. No part of this publication may be reproduced without prior consent of the Audit Office of New South Wales. The Audit Office does not accept responsibility for loss or damage suffered by any person acting on or refraining from action as a result of any of this material.

Parliamentary reference - Report number #353 - released (13 July 2021).

The Auditor-General for New South Wales, Margaret Crawford, has today released a report on Transport for NSW’s (TfNSW) acquisition of 4–6 Grand Avenue in Camellia.

This audit, which was requested on 17 November 2020 by the Hon. Andrew Constance MP, the Minister for Transport and Roads, examined:

• whether TfNSW conducted an effective process to purchase 4–6 Grand Avenue, Camellia
• whether TfNSW has effective processes and procedures to identify and acquire property required to deliver the NSW Government’s major infrastructure projects.

The audit found that TfNSW conducted an ineffective process when it purchased 4–6 Grand Avenue, Camellia. The audit also found that TfNSW’s internal policies and procedures to guide the transaction were, and continue to be, insufficient.

The Auditor-General has made seven recommendations to address the issues identified in the report.

On 17 November 2020, the Hon. Andrew Constance MP, the Minister for Transport and Roads, requested this audit under section 27B(3)(c) of the Public Finance and Audit Act 1983.

On 15 June 2016, Transport for New South Wales (TfNSW) acquired 6.3 hectares of land at 4–6 Grand Avenue, Camellia, by agreement from Grand 4 Investments Pty Ltd. Grand 4 Investments was a business entity established by the owners of Billbergia Pty Ltd, a property development and investment company.

TfNSW paid Grand 4 Investments $53.5 million and assumed liability for addressing environmental issues and contamination associated with the site. This took place seven months after the vendor acquired the land as part of a competitive Expression of Interest process, in which TfNSW also participated, for $38.15 million.

TfNSW is the NSW Government agency responsible for most major transport infrastructure projects in New South Wales. TfNSW acquired the Camellia site for use as a stabling and maintenance depot to support the Parramatta Light Rail (PLR) project.

Consistent with the minister’s request, this audit assessed:

• whether TfNSW conducted an effective process to purchase 4–6 Grand Avenue, Camellia
• whether TfNSW has effective processes and procedures to identify and acquire property required to deliver the NSW Government’s major infrastructure projects.

In considering the effectiveness of the processes for this purchase, the audit considered:

• the requirements of the Land Acquisition (Just Terms Compensation) Act 1991 (the Act)
• the application of sound processes to manage risk to the NSW Government and to achieve value for money
• the application of disciplines associated with complex procurement, such as probity, in a NSW Government context.

In 2015, TfNSW identified that it would require a stabling and maintenance depot in the Camellia area for the Parramatta Light Rail

In 2014, TfNSW commissioned an external engineering consultancy to undertake a feasibility design study for the Parramatta Light Rail - the Parramatta Transport Corridor Strategy Feasibility Design study (herein referred to as ‘the feasibility study’). In early 2015, TfNSW received the feasibility study, which was one of several key sources that informed the development of business cases for the PLR.

The feasibility study recommended that TfNSW should consolidate the maintenance and cleaning operations with overnight stabling facilities on one site. The study noted that the optimal location for any such site would be in close proximity to the proposed network, and noted that the site must have access to road connections to accommodate access for cars and trucks.

The study found that a centrally located stabling and maintenance facility would be required for all routes serving the Parramatta CBD, and that the Camellia industrial area was a preferred location for such a facility. The study noted that the Camellia area was contaminated.

The feasibility study notes that its conclusions were based on assumptions about the light rail system adopted and decisions made by the future operator of the system, who had not yet been selected or appointed.

TfNSW's decision to progress a potential acquisition in 2015 considered the risk that the site may not be required

TfNSW's FIC was responsible for making decisions on funding allocations at a whole of program level within TfNSW. FIC was also responsible for approving ‘high-risk/high-value’ variations to program budgets. Members of the FIC included:

• Secretary of Transport for NSW
• Deputy Secretary, Infrastructure and Services
• Deputy Secretary, Freight, Strategy and Planning
• Deputy Secretary, Customer Services
• Deputy Secretary Finance and Investment
• Deputy Secretary People and Corporate Services.

An April 2015 submission, from the then Deputy Director-General to the agency’s FIC, sought authorisation and funding approval to participate in an Expression of Interest sale process. It noted the risk that the project may not go ahead. The submission advised that:

By acquiring a strategic site now, it reduces the risk of having to pay an improved value or a value that may be subject to rapidly improving land values due to changes in land use and rezoning.

The property can be acquired for the project, held strategically and income generated by leasing the site as hardstand ¹ space until the project requires the land for the Parramatta Light Rail project.

If the project does not proceed in the medium to longer term, the property can be sold at a premium to what has been paid today as property fundamentals improve.

This submission acknowledged the risks associated with environmental contamination and proposed that these risks would be managed by negotiating a contract where the remediation and associated expenses would be at the landowner’s cost. 

TfNSW assessed the 4–6 Grand Avenue site as one of several sites in Camellia that was a feasible location for a stabling and maintenance facility

The Departmental feasibility study assessed six potential sites for a stabling and maintenance facility, including 4–6 Grand Avenue, noting strengths and weaknesses of each site. A different site on Grand Avenue was assessed as the ‘base case’ option (1 Grand Avenue). The study’s comments on the 4–6 Grand Avenue site included the following:

With an area of approximately 63,000m², this site has sufficient space for a depot with the required stabling yard and maintenance facilities. The location allows for good road access and LRT [light rail transit] access would be from Grand Avenue, which may require a road crossing or signalised intersection. The site has been used for general industrial uses; however the land has been cleared and is currently undergoing remediation ². The site is not affected by flooding based on one in 100-year flood data.

In early 2015, once the opportunity to acquire 4–6 Grand Avenue emerged, TfNSW commissioned a specific feasibility study of the 4–6 Grand Avenue site. The feasibility studies clearly documented the existence of environmental contamination. In April 2015, the report concluded:

Given the limitations of this report and within the parameters that have been set it is concluded that from a spatial and geographic perspective the site at 6 Grand Avenue would be suitable as a stabling and maintenance depot for the Parramatta light rail project. There are few engineering and environmental constraints that would affect the feasibility level analysis of this site and all issues identified, within this desk study, are considered to be resolvable. However this being said there is a significant amount of work necessary to reach the final layout and definition of the stabling and maintenance depot. There are numerous items which require further consideration and conformation; planning approvals could impose restrictions on building heights, noise mitigation measures, light and visual impact requirements all of which can have significant impacts on the spatial requirements of any stabling and maintenance depot. 

The acquisition of 4–6 Grand Avenue was not informed by a Property Acquisition Strategy

For major projects, TfNSW typically requires the project team to complete a Property Acquisition Strategy, which is intended to guide both process as well as specific acquisition issues expected to be faced during the project. The Property Acquisition Strategy is not a mandated document but is a recommended tool to support property acquisition as part of major projects.

TfNSW did not have a Property Acquisition Strategy in place to guide the 2015 Expression of Interest process. On 6 November 2015, the then Project Director for the PLR project emailed the property team, noting a need to develop a Property Acquisition Strategy to close off the scoping design and preliminary business case.

In January 2016, TfNSW developed a draft Property Acquisition Strategy for the Parramatta Light Rail Project, although it was silent on the potential sites for the stabling and maintenance facility.

TfNSW focussed on 4–6 Grand Avenue because it was available and aligned to TfNSW's strategic interests

In early 2015, officials commenced monitoring the market for industrial real estate in the Camellia area and surrounds for possible sites for a stabling and maintenance facility.

In March 2015, then owner of the site, Akzo Nobel Pty Limited released the 4–6 Grand Avenue site through an Expression of Interest process managed by CBRE.

TfNSW’s then Deputy Director-General, Planning, sought approval from FIC to lodge an Expression of Interest up to $30.0 million. Approval was sought on the basis that it would ‘provide certainty for the Parramatta Light Rail project by allowing for a depot site in a suitable location and potentially avoid higher costs or longer timeframes associated with compulsory acquisition following completion of the project’s business case’. FIC approved the request at its meeting on 9 April 2015.

At this time, TfNSW had not conducted any analysis of financial or operational benefits and costs of the potential sites identified in earlier feasibility studies. TfNSW staff advised us that the decision to participate in the Expression of Interest process for 4–6 Grand Avenue was because it was available. There is no documentation substantiating this statement, which TfNSW staff provided verbally as part of this audit.

In November 2015, TfNSW was advised that it was unsuccessful in the Expression of Interest process and that Grand 4 Investments (a related entity of Billbergia) had purchased 4–6 Grand Avenue. TfNSW did not conduct any further analysis of alternative potential sites in Camellia between this date and commencing discussions with Grand 4 Investments in April 2016. In that time there had been some movement on other properties that were included in the feasibility study, including 37–39a Grand Avenue being under offer in September 2015.

In March 2016, TfNSW approached CBRE to organise a meeting with Grand 4 Investments. On 1 April 2016, TfNSW met with Grand 4 Investments.

TfNSW advises that a perceived benefit of the 4–6 Grand Avenue site was that it was not subject to other uses or leaseholds that would increase the cost of compulsory acquisition. Officers involved in the acquisition advised that other nominated sites in the feasibility study were subject to other uses or leaseholds. 

Appendix one – Response from agency 

Appendix two – About the audit 

Appendix three – Performance auditing

Copyright Notice

© Copyright reserved by the Audit Office of New South Wales. All rights reserved. No part of this publication may be reproduced without prior consent of the Audit Office of New South Wales. The Audit Office does not accept responsibility for loss or damage suffered by any person acting on or refraining from action as a result of any of this material.

Parliamentary reference - Report number #349 - released (18 May 2021).

A report released today by the Auditor-General for New South Wales, Margaret Crawford found that select advertising campaigns conducted by Service NSW and the NSW Rural Fire Service met most requirements of the Government Advertising Act, regulations, Guidelines and other laws. However, the audit found that Service NSW inappropriately used its post campaign evaluation to measure sentiment towards and confidence in the NSW Government.  

While agency analysis shows that the ‘Cost of Living’ (phases 2 and 3)  and ‘How Fireproof is Your Plan?’ campaigns achieved most of their objectives, the campaign objectives and targets set by both agencies were not sufficient to measure all aspects of campaign effectiveness. 

The report makes two recommendations to the Department of Customer Service. The first is to review its guidance to ensure agencies are not using post campaign evaluations to measure sentiment towards the government. The second, to review its guidance and the new process of peer review to ensure they support agencies to comply with the Act, the regulations and the Guidelines. 

The Government Advertising Act 2011 requires the Auditor General to conduct an annual performance audit of one or more government agencies to see whether their advertising activities were carried out in an effective, economical and efficient manner and in compliance with the Government Advertising Act 2011.
 

Read full report (PDF)

The Government Advertising Act 2011 (the Act) requires the Auditor-General to conduct a performance audit on the activities of one or more government agencies in relation to government advertising campaigns in each financial year. The performance audit assesses whether a government agency or agencies have carried out activities in relation to government advertising in an effective, economical and efficient manner and in compliance with the Act, the regulations, other laws and the Government Advertising Guidelines (the Guidelines). This audit examined two campaigns run during the 2018–19 and 2019–20 financial years respectively:

• the 'Cost of Living' campaign run by Service NSW (phases 2 and 3 delivered in 2018–19)
• the 'How Fireproof Is Your Plan?' (Fireproof) campaign run by NSW Rural Fire Service (year two of a three-year campaign delivered in 2019–20).

Section 6 of the Act prohibits political advertising. Under this section, material that is part of a government advertising campaign must not contain the name, voice or image of a minister, member of parliament or a candidate nominated for election to parliament or the name, logo or any slogan of a political party. Further, a campaign must not be designed to influence (directly or indirectly) support for a political party.

In 2018–19, Service NSW delivered phases 2 and 3 of the 'Cost of Living' campaign. The Cost of Living advertising campaign aimed to build awareness of the help available to ease the cost of living for people under financial pressure including awareness of specific rebates that can be claimed. As part of the Cost of Living program, Service NSW developed a webpage designed as a single portal to access more than 40 NSW Government savings, rebates and initiatives (which originated from over 12 different agencies). It also launched the Cost of Living service which includes face to face meetings and phone interviews to help people claim rebates from the NSW Government. Phase 2 of the campaign ran from September 2018 to August 2019. Phase 3 of the campaign ran from January 2019 to July 2019. The budgets for phases 2 and 3 were $4.127 million and $934,800 respectively. See Appendix two for more details on this campaign.

Campaign materials we reviewed did not breach section 6 of the Act

The audit team reviewed campaign materials developed as part of the paid advertising campaign including radio transcripts, digital videos and display. The audit team did not review the use of social media outside paid social media content as section four of the Act defines government advertising as the dissemination of information which is funded by or on behalf of a government agency. See Appendix two for examples of campaign materials for this campaign.

Section 6 of the Act prohibits political advertising as part of a government advertising campaign. A government advertising campaign must not:

• be designed to influence (directly or indirectly) support for a political party
• contain the name, voice or image of a minister, a member of parliament or a candidate nominated for election to parliament
• contain the name, logo, slogan or any other reference to a political party.

The audit found no breaches of section 6 of the Act in the campaign material we reviewed. 

Post-campaign evaluations measured sentiment towards and confidence in the NSW Government

The post-campaign evaluation for phases 2 and 3 measured levels of confidence with the statement ‘the NSW Government has your best interests at heart’, despite the fact this was not a stated objective of the campaign. This is not an appropriate use of the post-campaign evaluation, which should measure the success of the campaign against its stated objectives. The post-campaign evaluation for phase 3 found that exposure to the campaign improved sentiment towards the government amongst those who did not have confidence in the NSW Government.

Service NSW advised that it was important to measure the sentiment of the advertising including the wording 'best interests' as it did not want the whole of government brand to be detrimental to customer engagement with applying for the rebates.

Following phase 2, Service NSW conducted analysis of media sentiment using the key words 'cost of living' and the names of the Premier, Treasurer and Minister for Customer Service. The analysis presented the level of positive, negative and neutral media sentiment. The Government Advertising Guidelines 2012 list the purposes that government advertising may serve which do not include improving the perception of the government. The inclusion of this analysis in Service NSW's post-campaign evaluation creates a risk that the results may be used for party political purposes.

Section 10 of the Act restricts agencies from carrying out a campaign after 26 January in the calendar year before the Legislative Assembly is due to expire and before the election for the Legislative Assembly in that year. Service NSW authorised a media agency to book media in line with the media plans for the campaign. The media plans for the campaign show that Service NSW did not authorise or plan to run any advertisements between 27 January 2019 and 23 March 2019.

Service NSW did not set targets for all rebates advertised in phase 2

Service NSW did not set targets for four of the seven rebates that were advertised as part of phase 2 of the campaign. These rebates were the Family Energy Rebate, Appliance Replacement Offer, National Parks Concession Offer and the Pensioner Travel Voucher. As a result, it was unable to evaluate whether the advertisements for these rebates were effective. Service NSW advised that at the time the campaign went to peer review, when campaign objectives are set, it did not know which rebates would be included in the advertisements.

Service NSW stated in its submission to the Department of Premier and Cabinet that it may change the creative content for phase 2 as it announced new initiatives and rebates. The peer review process should have ensured that Service NSW set targets for any additional rebates or savings it intended to advertise before that advertising commenced to ensure a strategic approach to the campaigns that clearly demonstrated anticipated benefits were in place.

The post-campaign evaluation for phase 2 shows that the advertising campaign met most of its objectives

Service NSW set overall campaign objectives and specific targets for some rebates advertised as part of phase 2 of the campaign. The objectives, targets and results for phase 2 are shown in Exhibit 5. In phase 2, Service NSW established baseline data on levels of awareness of government rebates during the peer review process. The baseline level of awareness for government rebates was 44 per cent. The level of awareness for specific rebates was 46 per cent for the Compulsory Third Party (CTP) green slip refund, and 21 per cent for both Active Kids and Toll Relief.

Post-campaign evaluation reports for phase 2 show that the campaign met its objective to raise awareness of NSW Government rebates, achieving a 16 per cent increase in awareness from 44 per cent to 51 per cent. The campaign did not meet its target to increase awareness of the CTP green slip refund by ten per cent.

Service NSW did not report the results of the uptake of the CTP green slip refund, Active Kids and Toll Relief in its post campaign effectiveness report submitted to the Department of Premier and Cabinet. However, other post-campaign evaluation documentation, which Service NSW advise was submitted to the Department of Premier and Cabinet, show that these targets were met.

Service NSW did not report to the Department of Premier and Cabinet on whether it achieved the target of a ten per cent increase of average monthly visits to the Cost of Living webpage. Service NSW reported that it had achieved an average of 11,753 visitors to the webpage per day during the campaign. These average daily results indicate that the target was met.

Exhibit 5: Phase 2 - campaign objectives, targets and results

Campaign objectives and targets	Does the post-campaign evaluation show that the target was met?
1. a) Increase awareness of rebates from the NSW Government by ten per cent.	Image
b) Increase average monthly visits to the Cost of Living webpage by ten per cent.	Image
2. Increase awareness of rebates and savings by ten per cent for:
CTP green slip refund	Image
Active Kids	Image
Toll Relief.	Image
3. Increase awareness that NSW Government initiatives relating to the cost of living are available via Service NSW by ten per cent.	Image
4. Increase the uptake of rebates and savings for the CTP green slip refund, Active Kids and Toll Relief by ten per cent.	Image

Key

Image

Yes

Image

Not Fully

The post-campaign evaluation for phase 3 shows that the advertising campaign met most of its objectives

Service NSW set overall campaign objectives and specific targets for the two rebates advertised as part of phase 3 of the campaign. The objectives, targets and results for phase 3 are shown in Exhibit 6.

In phase 3, Service NSW established baseline data on levels of awareness during the peer review process. The baseline level of awareness for government rebates was 44 per cent. This is the same baseline that was used to measure performance for phase 2 of the campaign. Service NSW did not set baselines for awareness and uptake of Energy Switch and Creative Kids as these were new services.

Post-campaign evaluation reports for phase 3 show that the campaign met its objective to raise awareness of NSW Government rebates by ten per cent, achieving a 30 per cent increase in awareness from 44 per cent to 57 per cent. The overall increase in message take-out was met with 43 per cent agreeing with the message that the NSW Government is taking steps to ease the cost of living. The campaign achieved awareness and uptake targets for the specific rebates included in phase 3, except for awareness of Creative Kids which achieved 28 per cent awareness, falling short of the 30 per cent awareness target.

Exhibit 6: Phase 3 - campaign objectives, targets and results

Campaign objectives and targets	Does the post-campaign evaluation show that the target was met?
1. Increase message takeout that ‘The NSW Government is taking steps to help ease the cost of living in NSW’ by ten per cent for those who can recall the campaign.
2. Increase awareness that the NSW Government has a range of rebates and savings by ten per cent.
3. Generate awareness with NSW residents aged 18+ of:
Energy Switch (15 per cent awareness)
Creative Kids (30 per cent awareness).
4. Create uptake of Energy Switch and Creative Kids (8,356 clicks on the Energy Switch website and 107,938 Creative Kids vouchers downloaded with 70 per cent conversion).

Key

Yes

Not Fully

The timing of campaign phases meant that it was difficult for Service NSW to evaluate each distinct campaign phase and reduced opportunities to incorporate learnings from previous phases

Service NSW commenced planning for phase 2 of the campaign while phase 1 was still underway. This limited the opportunity for Service NSW to incorporate learnings from phase 1 into phase 2. There was some overlap in the timing of phase 2 and the start of phase 3 of the campaign, making it difficult to evaluate the effectiveness of each distinct campaign phase. Both phases 2 and 3 had the same high-level outcome objective to raise awareness of rebates by ten per cent. The baseline measures that were used to evaluate performance for phase 3 were the same as those used to evaluate phase 2. As a result, Service NSW was not able to separately evaluate these two phases of the campaign. This is important given the budgets for phases 2 and 3 were $4.127million and $934,800 respectively.

Service NSW allocated 7.5 per cent of its media budget to communications with culturally and linguistically diverse (CALD) and Aboriginal audiences

The NSW Government CALD and Aboriginal Advertising Policy requires that agencies spend at least 7.5 per cent of an advertising campaign media budget on direct communications with CALD and Aboriginal audiences. Service NSW authorised a media company to book media in line with the media plans for the campaign. The media plans for phases 2 and 3 of the campaign indicate that Service NSW met this requirement, with 7.5 per cent of the budget allocated to these audiences in phase 2 and 10.4 per cent in phase 3.

The post campaign evaluation for phases 1 and 2 of the Cost of Living campaign contained a recommendation to look at other opportunities to reach CALD audiences. Effective communication with CALD audiences was particularly important in phase 3 of the campaign, where they made up 30 per cent of the target audience for the Creative Kids advertisement. The post-campaign analysis for phase 3 showed that the campaign performed well with some, but not all CALD audiences. The post-campaign analysis also showed low awareness and uptake with Aboriginal audiences. Pre-campaign focus groups in phase 3 found Aboriginal audiences had a negative reaction to the campaign tag line ‘NSW Government is helping with the cost of living’ however this tagline was still used in some advertisements in phase 3.

The cost-benefit analysis (CBA) for phase 2 did not accurately assess the benefits of the campaign and did not assess the costs and benefits of alternatives to advertising

Under the Government Advertising Act 2011, agencies are required to prepare a CBA when the cost of the campaign is likely to exceed $1 million. The CBA conducted by Service NSW for phase 2 includes $8 million in benefits attributed to the advertisements for the Energy Switch tool and $6.9 million in benefits attributed to the advertisements for Creative Kids vouchers. These benefits should not have been included in the CBA for phase 2 as they were not included in this phase of the campaign. The CBA did not estimate the benefits of some other rebates and savings advertised in phase 2 of the campaign. This means that the CBA did not accurately assess the benefits of the campaign. Service NSW advised that at the time the CBA was developed it had not selected the rebates to be included in the campaign.

The Government Advertising Guidelines require agencies to consider options other than advertising to achieve the desired objective including a comparison of costs and benefits. The CBA developed as part of phase 2 identified using existing NSW Government communication channels as an alternative to advertising but did not assess the costs and benefits of this alternative.

This is a repeat finding from two previous government advertising audits. The report ‘Government Advertising: 2015–16 and 2016–17’ found that both agencies subject to the audit did not meet the requirements in the guidelines to consider alternatives to advertising. The report made a recommendation to the Department of Premier and Cabinet to work with Treasury to ensure the requirements of the guidelines are fully reflected in the 'Cost-Benefit Analysis Framework for Government Advertising and Information Campaigns'. The report ‘Government advertising 2017–18’ found that one agency subject to the audit did not identify to what extent the benefits could be achieved without advertising, nor did it consider alternatives to advertising which could achieve the same impact as the advertising campaign.

Service NSW negotiated with a single creative agency in phase 2, making it difficult to demonstrate value for money

Agencies are required to obtain three quotes when procuring a creative agency on the prequalification scheme if the estimated cost of the creative content is greater than $150,000. In phase 2 of the campaign, Service NSW extended the contract with the creative agency used for phase 1 of the campaign and did not obtain three quotes despite the cost of the creative content for phase 2 being $731,480. The requirement to obtain three quotes was met in phase 1 when initially selecting this creative agency.

Service NSWs procurement policy details that direct negotiation may be appropriate where there is a compelling reason to renew or rollover a contract beyond temporal or convenience reasons or in the cases of a genuine emergency. In its briefing to the Chief Executive, Service NSW stated that this contract extension was sought due to the time-sensitive nature of the project and that if work was delayed by a tender process, Service NSW may not be able to meet marketing milestones and this could result in limited customer uptake. This reason is not a genuine emergency and is not compelling as it does not explain what consequences would occur if it did not meet the marketing milestones or if there was limited customer uptake.

Service NSW's procurement policy also states that under no circumstances must Service NSW employees enter into discussions with a supplier until the delegate has formally made their decision to enter into direct negotiation. Service NSW briefed the Chief Executive of Service NSW in relation to extending the contract on 5 September 2018. The briefing states that the creative agency had already begun developing creative content for phase 2 and Service NSW had already received quotes from the creative provider for the proposed work prior to 5 September 2018. Procurement sign-offs were not completed until 7 September 2018. The engagement of the creative provider prior to appropriate approvals was contrary to Service NSWs procurement policy.

The economy of the campaign may have been limited by not meeting the procurement requirements in phase 2. It is possible that the creative provider may have offered a more competitive rate if it was aware that Service NSW was seeking quotes from other creative providers. Additionally, it is possible that another creative provider could have provided better value for money.

In phase 3 of the campaign, the estimated cost of the creative exceeded $150,000 however Service NSW chose to contract two different creative agencies, and the cost for each agency fell below the threshold to obtain three quotes. Agencies are permitted to obtain one quote when using a creative provider on the prequalification scheme if the cost is between $50,000 to $150,000. Service NSW advised that it contracted two creative providers as two different project teams were responsible for the rebates, each with separate marketing budgets.

Service NSW allowed sufficient time for cost-efficient media placement

During the peer review process, the Department of Premier and Cabinet advised agencies about the time they should allow to ensure cost-efficient media placement. For example, the Department of Premier and Cabinet advised that agencies book television advertising six to 12 weeks in advance and that agencies book radio advertising two to eight weeks in advance.

Service NSW allowed sufficient time between the completion of the peer review process and the commencement of the first advertising. Service NSW signed the agreement with the approved Media Agency Services provider with sufficient time to achieve cost-efficient media placement for all types of media used in this campaign.

The campaign may have been misleading for some people who were not eligible for rebates

Advertisements we reviewed focused on the amount of savings that could be obtained from rebates, for example ‘Save up to $285’, and ended with a statement ‘To save, visit service.nsw.gov.au. This directed viewers to the Cost of Living website which contains eligibility information. However, the advertisements in phases 2 and 3 we reviewed did not contain any details on the eligibility for these rebates and not all advertisements stated that eligibility criteria apply. Service NSW advised that the eligibility criteria for each rebate is extensive and that it was not possible to include this in the creative material.

Post-campaign evaluations in phase 3 recommended that advertisements for Creative Kids should indicate eligibility (e.g. age criteria) as statements on savings have the potential to be misleading when not all viewers will be eligible for rebates. Social media analysis conducted following phase 2 showed ineligibility or inability to claim rebates or refunds caused anger for some respondents.

Some advertisements in phase 2 stated ‘we've got something for everyone’. However, as rebates were subject to eligibility criteria, it is possible that some residents in NSW would not be eligible for any rebates as part of the Cost of Living initiative. As such, this statement has the potential to be misleading.

The campaign included statements that underestimated the savings that some customers could obtain

The Guidelines require accuracy in the presentation of all facts, statistics, comparisons and other arguments. The Guidelines also require that all claims of fact included in government advertising campaigns must be able to be substantiated.

In phase 2, the possible savings customers could obtain for two rebates or savings exceeded the amounts stated in the advertising campaign. Exhibit 7 shows some advertisements in phase 2 which stated, ‘My Green Slip Saving Save up to $60’. However, the State Insurance Regulatory Authority website shows that savings for some types of motor vehicles under the 2017 CTP scheme exceed $60. The State Insurance Regulatory Authority website states that the average saving under this scheme has been $129. Service NSW advised that these advertisements were designed for regional markets and that it used different advertisements for metropolitan areas which contained different amounts of savings.

Some advertisements in phase 2 stated, ‘My Toll Relief save up to $700’. The Service NSW website states that drivers can obtain free vehicle registration if they have spent $1,352 or more in tolls in the previous financial year. The cost of registration for some vehicles exceeds $700. This means the savings detailed in the advertisement were lower than what some customers could actually save.

NSW Rural Fire Service conducted the 'How FireProof Is Your Plan?' (Fireproof) campaign. The Fireproof campaign is a three-year campaign which ran in 2018–19 (year one), 2019–20 (year two) and is planned for 2020–21 (year three). This audit examined year two of the campaign (2019–20).

The Fireproof campaign is a public safety campaign encouraging people to plan and prepare for bush fires across the summer period. The campaign aims to improve the quality of bush fire planning and preparation in the community and decrease the impact of fires on the community when they occur.

Campaign materials we reviewed did not breach section 6 of the Act

The audit team reviewed campaign materials developed as part of the paid advertising campaign for example radio advertisements, television commercials and digital displays. The audit team did not review the use of social media outside paid social media content as section four of the Act defines government advertising as the dissemination of information which is funded by or on behalf of a government agency. Examples of campaign materials are shown in Appendix two.

Section 6 of the Act prohibits political advertising as part of a government advertising campaign. A government advertising campaign must not:

• be designed to influence (directly or indirectly) support for a political party
• contain the name, voice or image of a minister, a member of parliament or a candidate nominated for election to parliament
• contain the name, logo, slogan or any other reference to a political party.

The audit found no breaches of section 6 of the Act in the campaign material we reviewed. 

NSW Rural Fire Service did not set targets for the second year of the campaign

The second year of the Fireproof campaign (2019–20) had the same objectives as the first year of the campaign (2018–19), however no specific targets were set for the second year. The advertising submission for the first year of the campaign (2018–19) details the targets for each objective as an increase of ten per cent against the baseline data to be achieved by March 2021, at the end of the three-year campaign.

The second year of the Fireproof campaign (2019–20) was one of the first campaigns approved under the new budget and peer review processes introduced by the Department of Customer Service in 2019–20. The new process for peer review introduced a new template for campaign submissions. The former template for campaign submissions contained more prompts for agencies to ensure the submission contained sufficient detail of campaign objectives, baseline measures, targets, dates for measurement and detail on how they would measure objectives. Despite this, the peer review process should have identified that NSW Rural Fire Service did not set targets for the second year of the campaign.

The 2016 Guidelines for Implementing NSW Government Evaluation Framework for Advertising and Communications requires campaign objectives to be SMART (specific, measurable, achievable, realistic and timed). NSW Rural Fire Service did not meet this requirement for year two of the Fireproof campaign.

Post-campaign evaluations showed increases against four out of five objectives, however there were no specific targets

NSW Rural Fire Service set three campaign objectives at the time it submitted the second year of the campaign (2019–20) to the Department of Customer Service for peer review. However, the post-campaign effectiveness report submitted to the Department of Customer Service measured campaign effectiveness against five campaign objectives. The objectives in the post-campaign effectiveness report were the same objectives set for the first year of the campaign, which is appropriate as this was a repeat campaign.

NSW Rural Fire Service achieved increases against four of their five objectives. However, as noted above there were no specific targets (such as percentage increases) against which performance of the 2019–20 campaign could be measured. Despite this, at the end of the second year, the Fireproof campaign had already achieved some of the targets that NSW Rural Fire Service had set for the end of the third year of the campaign. The post-campaign research showed that both audience recall and exposure to the campaign increased significantly from the prior year. The campaign objectives and results are shown in Exhibit 8.

For those people who already have a bush fire plan, the campaign aimed to increase the number of those plans which have included two or more elements from the Guide to Making a Bush Fire Survival Plan. Elements from the Guide to Making A Bush Fire Survival Plan include actions such as deciding what to take with you if you leave, ensuring you have the right equipment for defending your home and allocating responsibilities to members of a household. The post-campaign evaluation showed that the campaign did not achieve an increase against this objective for people who planned to stay and defend their property rather than leave.

Exhibit 8: Campaign objectives and results

Campaign objectives	Does the post-campaign evaluation show increases against the objective?
1. Continue to increase the number of people that have discussed and/or written a plan with regards to what they will do in the event of a fire.
2. Of those who indicate they have a plan, increase the number of people who have included two or more elements from the Guide to Making a Bush Fire Survival Plan:
for those who plan to leave
for those who plan to stay and defend.
3. Increase the frequency in completing preparation activities around a person’s property.
4. Increase the number of people who correctly assess it is their responsibility to complete preparation activities and enact their plan without direct intervention from emergency services.
5. Visits to MyFirePlan website.

Key

Yes

No

NSW Rural Fire Service achieved cost efficiencies by reusing creative content developed in the first year of the campaign

Total creative and production costs incurred in year one of the campaign were $1.08 million. Rather than commissioning new creative materials, NSW Rural Fire Service re-used the same creative content in year two of the campaign. NSW Rural Fire Service incurred $100,000 in creative and production costs in year two of the campaign and achieved cost-efficiencies by reusing the same creative developed in the prior year.

NSW Rural Fire Service allowed sufficient time for cost-efficient media placement and received free media placements

The Department of Customer Service advises agencies to work with media contacts to book media in advance to ensure a cost-efficient placement. Prior to 2019–20, the Department of Premier and Cabinet provided suggested timeframes for agencies to book media as part of the peer review process. For example, it advised agencies to book television six to 12 weeks in advance and book radio advertising two to eight weeks in advance. NSW Rural Fire Service allowed sufficient time for a cost-efficient media placement.

NSW Rural Fire Service received $4 million of free advertising time and space donated by media companies due to the extent and impact of the 2019–20 fire season.

The cost benefit analysis (CBA) did not assess the costs and benefits of alternatives to advertising

Under the Government Advertising Act 2011, agencies are required to prepare a CBA when the cost of the campaign is likely to exceed $1 million. As part of the CBA, the Government Advertising Guidelines require agencies to consider options other than advertising to achieve the desired objective including a comparison of costs and benefits.

The CBA for the Fireproof campaign (year two) notes that the proposed campaign is one component of a broader community engagement strategy which has been developed over time and is based on research and evaluation. The CBA considers two options to achieve the objectives of the campaign. The first option is community engagement activities without an advertising campaign and the second option is community engagement activities alongside an advertising campaign. The CBA does not identify and assess the costs and benefits of both of the options in order to assess the most cost-efficient option.

This is a repeat finding from two previous government advertising audits. The report ‘Government Advertising: 2015–16 and 2016–17’ found that both agencies subject to the audit did not meet the requirements in the guidelines to consider alternatives to advertising. The report made a recommendation to the Department of Premier and Cabinet to work with Treasury to ensure the requirements of the guidelines are fully reflected in the 'Cost-Benefit Analysis Framework for Government Advertising and Information Campaigns'. The report ‘Government advertising 2017–18’ found that one agency subject to the audit did not identify to what extent the benefits could be achieved without advertising, nor did it consider alternatives to advertising which could achieve the same impact as the advertising campaign.

Appendix one – Responses from agencies

Appendix two – About the campaigns

Appendix three – About the audit

Appendix four – Performance auditing

Copyright notice

© Copyright reserved by the Audit Office of New South Wales. All rights reserved. No part of this publication may be reproduced without prior consent of the Audit Office of New South Wales. The Audit Office does not accept responsibility for loss or damage suffered by any person acting on or refraining from action as a result of any of this material.

Parliamentary reference - Report number #342 - released 19 November 2020

This report examines whether NSW Health effectively planned and delivered major capital works to meet the demand for health services in New South Wales.

The report found that NSW Health has substantially expanded health infrastructure across New South Wales since 2015. However, the program was driven by Local Health District priorities without assessment of the State’s broader and future‑focussed health requirements.

The report found that unclear decision making roles and responsibilities between Health Infrastructure and the Ministry of Health limited the ability of NSW Health to effectively test and analyse investment options.

Project delays and budget overruns on some major projects indicate that Health Infrastructure's project governance, risk assessment and management systems could be improved.

The Auditor‑General recommends that NSW Health ensure its capital projects offer the greatest value to New South Wales by establishing effective policy guidance and enhancing project governance and management systems.

Read full report (PDF)

Since 2011–12, NSW Health has aimed to improve its facilities and build 'future focused' infrastructure. The NSW Government’s 2015–16 election commitments established a four-year $5.0 billion capital program for NSW Health to build and upgrade more than 60 hospitals and health services. The 2019–20 State Budget committed a further $10.1 billion over four years for another 29 projects. This is the largest investment to date on health capital works in New South Wales.

Recent reviews of infrastructure have recognised that population and demographic growth will require a change in the delivery and composition of health infrastructure, including considering greater use of non-traditional, non-capital health service options and assets.

To ensure that expenditure on capital works represents the best value for money, NSW Health's business cases need to be robust and supported by evidence that demonstrates they are worthy investments. The NSW Process of Facility Planning has been the main framework guiding the detailed planning and development of NSW Health's capital works proposals. This framework was developed by the then NSW Department of Health in 2010. Its aim is to ensure investment proposals are supported by rigorous planning processes that address health service needs and provide value for money.

Infrastructure projects of the complexity and scale being delivered by NSW Health carry inherent risks. For example, unplanned cost escalations can potentially impact on the State’s finances. Unforeseen delays can also reduce the intended benefits. The growth in the State’s health capital spend and project profile, means its exposure to such risks has increased over time.

The objective of this audit was to assess the effectiveness of planning and delivery of major capital works to meet demand for health services in New South Wales. To address this objective, the audit examined whether:

• the Ministry of Health has effective procedures for planning and prioritising investments in major health capital works
• Health Infrastructure develops robust business cases for initiated major capital works that reliably inform government decision making
• Health Infrastructure has effective project governance and management systems that support delivering projects on-time, within budget and achievement of intended benefits.

The audit focused on the Ministry of Health and Health Infrastructure – being the lead agencies within NSW Health responsible for prioritising, planning and delivering major health capital works across the State. The audit examined 13 business cases for eight discrete projects over a ten-year period.

Over the period of review, NSW Government policies for business case development and submission have emphasised that effective governance arrangements are critical to a proposal's successful implementation.

NSW Health's Process of Facility Planning similarly highlights the importance of effective governance and project management for achieving good outcomes. It prescribes a general governance structure managed by Health Infrastructure that can be tailored to the planning and delivery of health infrastructure projects greater than $10.0 million.

The three major hospital redevelopments examined in metropolitan, regional and rural areas had a combined Estimated Total Cost of more than $1.2 billion and comprised eight discrete projects and 13 separate business cases.

Almost all these projects experienced delivery challenges which impacted achievement of their original objectives and intended benefits. This is expected in complex and large-scale health infrastructure programs. However, in some projects the impacts were significant and resulted in substantial delays, unforeseen costs, and diversion of resources from other priority areas.

Our review of the selected case studies highlighted opportunities for enhancing governance and project management. Specifically, it indicates a need for improving transparency in the management of contingencies, risk management and assessments particularly relating to adverse site conditions and the selection of contractors. There is also a need to strengthen forward planning for options to address unfunded priorities within business cases that risk complicating the delivery of future project stages resulting in unforeseen costs and potentially avoidable budget overruns.

In February 2017, the Ministry's Capital Strategy Group approved the use of surplus funds of $13.76 million from Stage 1 of the Hornsby Ku-ring-gai Hospital Redevelopment for new works deemed needed to support Stage 2. Following this decision, Health Infrastructure finalised and submitted a business case addendum for Stage 1 to the Ministry in March 2017, addressing the new works comprising a two-storey building for medical imaging and paediatric floors. The business case addendum also addressed options to fit out and procure major medical imaging equipment. The Ministry approved the Stage 1 business case in July 2017, noting the Ministry's Capital Strategy Group had already approved the use of remaining Stage 1 funds to deliver the new works.

Stage 1 was completed in 2015, almost two years before the Stage 1 business case addendum was prepared in February 2017.

The Ministry's decision to approve the new works using $13.76 million of surplus Stage 1 funds did not comply with the NSW Treasury Circular TC 12/20. This policy establishes the Treasurer's approval must be sought and received before a new capital project with an Estimated Total Cost of $5.0 million or more can be approved by NSW Health. The Ministry therefore exceeded its delegated authority in making this decision, as it was not evident it had sought and received the Treasurer's approval prior to doing so.

Consequently, the surplus Stage 1 funds should not have been used by the Ministry to deliver new works in the circumstances. Instead, they should have been released from the Stage 1 project in accordance with established NSW Health procedures, and the Stage 1 Estimated Total Cost revised down accordingly. This did not occur, and NSW Health ultimately directed $11.0 million in surplus Stage 1 funds to the new works.

These circumstances indicate a need to strengthen transparency and accountability within NSW Health for the approval of new projects, and how contingency funds are used in the management of major health capital works. They also demonstrate the impact of weaknesses with options appraisal as the initial Stage 1 business case did not consider alternative options for addressing the initially unfunded works later covered by the Stage 1 business case addendum and ultimately funded from the Stage 1 contingency provision.

In addition to proposing the above-noted new works, the 2017 Stage 1 Business Case Addendum for the Hornsby-Ku-ring-gai development sought to retrospectively address the estimated funding gap of around $14.0 million for the internal fit out, supply of major medical imaging equipment, and cost to operate the medical imaging service at Hornsby Ku-ring-gai Hospital also not addressed in the originally Stage 1 business case.

The Stage 1 business case addendum considered various procurement options to purchase and run the medical imaging services ranging from State operation purchase options to private operation purchase options.

It recommended outsourcing the operation and provision of equipment to the private sector based on estimated savings to the public sector initially of around $650,000 per annum reducing over time to $270,000. The Ministry endorsed this option in June 2017, but it did not ultimately proceed.

A July 2018 report to the Executive Steering Committee on the project shows NSW Health later decided to deliver operation of the medical imaging unit 'traditionally' with an updated estimate of the cost at approximately $16.4 million. The report also shows the Ministry supported the costs now being met by the Northern Sydney Local Health District.

This means the funding gap previously identified in the Stage 1 business case addendum for fitting out the medical imaging building and supply of major medical equipment would need to be met fully by the State, representing a $16.4 million cost overrun for the project.

Examined reports to the Executive Steering Committee show this was largely funded by the Northern Sydney Local Health District via the disposal of land realising approximately $15.0 million in proceeds.

This initially unforeseen cost, along with the additional $11.0 million for the new works approved under the Stage 1 business case addendum, were ultimately merged with the Stage 2 project initially approved in 2017–18 with an Estimated Total Cost of $200 million.

The 2019–20 State Budget provided an additional $65.0 million for a further Stage 2A to deliver additional built capacity to support outpatient services, enhanced allied health services, re-housed community health services and the delivery of prioritised clinical services unfunded as part of Stage 2. The funds were approved based on an Investment Decision Template (IDT) that examined two options in addition to the base case representing scoping alternatives to the preferred master planned capital solution.

However, we found the IDT showed around 23 per cent of the $65.0 million sought (i.e. $15.0 million) was to be allocated to fund the deficit in Stage 2, which had arisen as a result of project delays due to adverse site conditions. This was not discussed in the IDT.

The February 2020 report to the Executive Steering Committee shows a combined Stage 2 and 2A final forecast cost of $292.6 million against a potential budget of $290.7 million representing an overall deficit for the project of around 0.6 per cent.

However, this favourable final budget position does not transparently show the funding challenges experienced over the project's implementation to-date. The three major budget issues include:

• inappropriate use of around $11.0 million in Stage 1 contingency for originally unfunded works contrary to Treasury policy
• the additional $16.4 million cost unforeseen in the Stage 1 business case for delivering medical imaging services mostly funded through the sale of land
• an additional $15.0 million from Stage 2A to cover the budget overrun in Stage 2 due to adverse site conditions.

The cumulative impact of these events is that Stages 1 and 2 of the Hornsby project cost approximately $42.4 million than it should have in the circumstances around 14 per cent more than what the revised combined Estimated Total Cost for both stages should have been after releasing the $11.0 million in surplus Stage 1 funds, with Stage 2 delayed by around 14 months.

Major construction projects often experience adverse site conditions which can be difficult to fully detect in advance. However, we found this was a common occurrence in the projects we examined sometimes with significant time and/or budget impacts indicating scope to enhance related risk and cost assessments. Specifically:

• Hornsby Ku-ring-gai Hospital Redevelopment Stage 2: adverse site conditions during demolition works resulted in an 11-month delay for delivering the medical imaging unit and 14-month delay completing Stage 2 main works including need for additional $15.0 million in funds to cover the resultant budget deficit for the project.
• Blacktown Mt Druitt Hospital Redevelopment Stage 2: adverse site conditions combined with project complexity delayed completion of the early works by approximately five months. This contributed to the delay in completing the main construction works which occurred around nine months later than planned in the business case.
• Dubbo Health Service Redevelopment Stages 3 and 4: Health Infrastructure advised adverse site conditions including asbestos containing materials and ground conditions delayed works for the main building with completion forecast for March 2021, around 21 months later than planned in the final business case. This resulted in the need for additional $13.5 million to cover increased construction costs and risks, increasing the Stage 3 and 4 forecast final cost from $150 million to $163.5 million as at February 2020.

These examples indicate a risk the cumulative impact of adverse site conditions may be substantial when measured across both time and Health Infrastructure's full delivery program. They also point to potential for Health Infrastructure to achieve efficiencies and improved outcomes from strengthening its approach to assessing and mitigating the risks from adverse site conditions.

Main construction works on Stage 1 of the Dubbo Health Service Redevelopment were completed in October 2015, approximately 13 months later than planned in the final business case. Delays were mainly due to insolvency of the early works contractor resulting in their departure from the project. The ensuing 11-month delay in completing the early works significantly impacted the overall schedule and delivery of main construction works.

The insolvency event was significant as it affected nine separate Health Infrastructure projects – three of which had yet to reach practical completion. It also affected state-funded projects in other sectors. It resulted in the need for additional funding of $11.5 million that was provided in the 2014–15 State Budget increasing the total Stage 1 and 2 budget from $79.8 million to $91.3 million.

Health Infrastructure’s analysis of lessons learned shows it worked actively to mitigate the impacts of the insolvency event across all affected projects. However, it also indicates a risk the lessons were mainly focused on mitigating the impacts after an insolvency event occurred rather than on prevention.

Although Health Infrastructure initially commissioned a financial assessment of the now insolvent early works contractor before engagement, it did not detect any risks of the impending insolvency and instead concluded the contractor was in a strong financial position. However, the contractor became insolvent shortly after commencement approximately seven months later. This indicates a risk of weaknesses in the assessment performed that was not explicitly addressed by the lessons learned.

Delivery of the main construction works were further impacted by disputes with the main works contractor over the scope of works for the renal unit resulting in Health Infrastructure terminating the contract in November 2016 following lengthy negotiations over several months.

The scope of works relating to the renal unit were ultimately transferred to Stages 3 and 4 and were delivered in December 2019, around five years later than originally planned in the business case.

Health Infrastructure advised the delay was ultimately beneficial to the project because the refurbishment works for the renal unit, initially scheduled for Stages 1 and 2, would have been demolished to accommodate the new Western Cancer Centre proposed after Stages 1 and 2 and currently being delivered in parallel with Stages 3 and 4.

Health Infrastructure advised the actual cost of Stages 1 and 2 was $84.7 million against the budget of $91.3 million. The residual $6.6 million relates to the renal works not delivered during Stage 1 and 2 and transferred to Stage 3 and 4.

Health Infrastructure advised the contractual provisions for mitigating insolvency events 'in-flight' are limited highlighting the importance of proactive and effective due diligence prior to engaging contractors for significant construction projects.

Health Infrastructure's 2017-20 Corporate Plan identifies the development of a quality framework to support delivery of future-focused outcomes as a key organisational priority. Related initiatives within the Corporate Plan describe a framework underpinned by a Quality Committee providing advice on:

• records management, to meet the requirements of the State Records Act 1998
• project assurance, to ensure future focused outcomes and enhance Health Infrastructure's Standards, Policies, Procedures and Guidelines, Templates and Design Guidance Notes
• knowledge management and library services, to promote and leverage from project learnings.

Although Health Infrastructure has some elements of a quality framework it is not yet fully in place. Health Infrastructure advised it had yet to establish the quality framework and related committee described in its Corporate Plan due in part to its focus on responding to the growth of its capital program.

Health Infrastructure's Development and Innovation team has been active in supporting continuous improvement in knowledge and project management including development of business cases. Although useful, these initiatives have relied heavily on leveraging and disseminating insights from Gateway reviews and have not formed part of a systematic quality and continuous improvement framework.

The limited focus on the quality of business cases is reflected in internal performance monitoring and reporting which focuses mainly on tracking the delivery of projects against internal benchmarks, often revised from the baselines in the business case, and expenditure against cashflow targets. There is no evident internal monitoring and/or reporting to the Chief Executive and Board on defined quality metrics linked to business case development and staff capability.

Performance reporting on balanced scorecard metrics has similarly focused mainly on process rather than quality and has been inconsistent in recent years.

Appendix one – Response from agency

Appendix two – About the audit

Appendix three – Performance auditing

Appendix four – Ministry of Health planning tools and guidelines

Appendix five – Streamlined investment decision process for Health Capital Projects

Appendix six – Timeline of business cases and relevant policy guidelines

Copyright notice

© Copyright reserved by the Audit Office of New South Wales. All rights reserved. No part of this publication may be reproduced without prior consent of the Audit Office of New South Wales. The Audit Office does not accept responsibility for loss or damage suffered by any person acting on or refraining from action as a result of any of this material.

Parliamentary reference - Report number #338 - released 12 August 2020