Refine search

Reports

18 December 2018

Published

Government Advertising 2017-18

Premier and Cabinet

Compliance

Regulation

The State Insurance Regulatory Authority’s (SIRA) ‘green slip refund’ campaign, and the TAFE semester one 2018 student recruitment campaign, complied with most requirements of the Government Advertising Act 2011 and the Government Advertising Guidelines, according to a report released today by the Auditor-General for New South Wales, Margaret Crawford.

The Government Advertising Act 2011 (the Act) requires the Auditor-General to conduct a performance audit on the activities of one or more government agencies in relation to government advertising campaigns in each financial year. The performance audit assesses whether a government agency or agencies has carried out activities in relation to government advertising in an effective, economical and efficient manner and in compliance with the Act, the regulations, other laws and the Government Advertising Guidelines (the Guidelines).

This audit examined two campaigns conducted in 2017–18:

the 'Green slip refund' campaign run by the State Insurance Regulatory Authority (SIRA)
the semester one component of the 'TAFE NSW 2018 Student Recruitment Annual Campaign Program' run by the NSW TAFE Commission (TAFE).

Section 6 of the Act prohibits political advertising. Under this section, material that is part of a government advertising campaign must not contain the name, voice or image of a minister, member of parliament or a candidate nominated for election to parliament or the name, logo or any slogan of a political party. Further, a campaign must not be designed to influence (directly or indirectly) support for a political party.

Conclusion

Neither campaign breached the prohibition on political advertising contained in section 6 of the Act. Both campaigns also complied with most requirements of the Act, the regulations, other laws and the Guidelines. Neither agency could demonstrate that their campaigns were fully effective or economical.

SIRA did not breach section 6 of the Act, which prohibits political advertising. However, SIRA used its post-campaign evaluation to ask the public whether they believe the government was helping to reduce the cost of living by making reforms in a variety of areas, including some that were not related to the green slip campaign. SIRA advised that these additional statements were included to provide a broader context for any change in the green slip campaign survey results. This is not an appropriate use of the post-campaign evaluation because the post-campaign evaluation should measure the success of the campaign against its stated objectives.

Neither campaign met all their key objectives, limiting the overall effectiveness of the campaigns. SIRA successfully increased awareness of the availability of green slip refunds and met the target for the proportion of people claiming their refunds online. However, it did not meet its objective to inform the public about the reforms to the green slip scheme, beyond the refunds available to motorists. While 62 per cent of surveyed people were aware of the reforms, there was little knowledge about many specific aspects of the reforms, which people largely associated with lower insurance prices and refunds. TAFE was successful in achieving targets for changing the public perception of TAFE. However, it failed to achieve its semester one enrolment target.

SIRA was not able to demonstrate that its campaign was economical as it directly negotiated with a single supplier for the campaign's creative materials. This is contrary to the NSW Government's and SIRA's own procurement guidance that advise it to seek quotes from suppliers on a prequalification scheme if available. SIRA had access to the Advertising and Digital Communication Services prequalification scheme, but still continued with direct negotiations. While SIRA sought to demonstrate value for money by comparing the supplier's quote to the expenditure on creative materials in other campaigns, it did not document this evaluation to ensure that decision makers were fully informed.

TAFE was not able to demonstrate that its campaign was economical as it did not compare the campaign with a zero-advertising scenario to demonstrate the exact benefits directly attributable to the campaign. TAFE's cost-benefit analysis also did not identify to what extent benefits could be achieved without advertising, nor did it consider alternatives to advertising which could achieve the same impact as the advertising campaign. All these elements should have been included in TAFE's cost benefit analysis.

Both agencies achieved some efficiencies in implementing their campaigns. SIRA booked all of its media placements in a cost-efficient manner. TAFE booked most of its media placements in a cost-efficient manner and achieved further efficiencies through the re-use of previous campaign material.

The State Insurance Regulatory Authority (SIRA) conducted the 'Green slip refund' campaign between March and June 2018. SIRA ran this campaign to raise awareness of the Compulsory Third Party (CTP) refunds and reforms after the Motor Accidents Injuries Act 2017 commenced in December 2017. SIRA's view is that the reforms include a reduced cost for CTP insurance, benefits for at-fault drivers, reduced opportunity for fraud and attempts to lower insurance company profits. Green slip holders are also able to claim partial refunds on their 2017 green slip insurance premium. The campaign aimed to make green slip holders aware of the refunds available, encourage them to claim online and to inform people about the changes to the green slip scheme. The campaign focused on the first two of these objectives. The total cost of the campaign was $1.9 million. See Appendix two for more details on this campaign.

The 'Green slip refund' advertising campaign did not breach section 6 of the Act which prohibits political advertising. However, SIRA used its post-campaign evaluation to ask the public whether they believe the government was helping to reduce the cost of living by making reforms in a variety of areas, including some that were not related to the green slip campaign. SIRA advised that these additional statements were included to provide a broader context for any change in the green slip campaign survey results. This is not an appropriate use of the post-campaign evaluation because the post-campaign evaluation should measure the success of the campaign against its stated objectives.

The campaign met most of its objectives, including raising awareness of the green slip refunds and encouraging people to claim online. However, the campaign was not fully effective because it did not inform the public of the green slip reforms. This was one of the objectives of the campaign. Sixty-two per cent of people in the post-campaign survey stated that they were aware of the reforms, an increase from the baseline of 20 per cent. However, these people largely associated the reforms with lower insurance prices and had a low awareness of any other elements of the reforms, such as SIRA's view that the reforms introduced better support for people injured on the road. This indicates that the campaign did little to inform people about the green slip reforms beyond the price of insurance.

SIRA was able to ensure cost-efficient media purchases by signing its media booking authority within the timeframe advised by DPC.

SIRA could not demonstrate that the campaign was carried out economically. SIRA directly negotiated with a single supplier to procure the creative materials for this campaign. Direct negotiations make it difficult to ensure value for money due to the lack of competition. SIRA proceeded with direct negotiations despite being able to access a prequalification scheme which could increase competition. In doing so, SIRA did not follow government's or its internal procurement guidance. While SIRA sought to demonstrate value for money by comparing the supplier's quote to the expenditure on creative materials in other campaigns, it did not document this evaluation to ensure that decision makers were fully informed.

Campaign materials we reviewed did not breach section 6 of the Act

Section 6 of the Act prohibits political advertising as part of a government advertising campaign. A government advertising campaign must not:

be designed to influence (directly or indirectly) support for a political party
contain the name, voice or image of a minister, a member of parliament or a candidate nominated for election to parliament
contain the name, logo, slogan or any other reference to a political party.

The audit team found no breaches of section 6 of the Act in the campaign material we reviewed.

Before the start of the campaign, SIRA conducted a survey which asked people whether they agreed ‘that the NSW Government is helping to reduce the cost of living by making positive reforms to:

reduce the cost of green slips
reduce the cost of health insurance
increase the number of jobs
increase investment in the state.'

SIRA's initial submission to peer review listed one of the campaign objectives as improving the perception of the government as a positive reformer. DPC advised SIRA that this should not be included. SIRA removed this objective.

Even though SIRA appropriately removed this objective, the post-campaign evaluation still measured agreement with the above statements, three of which did not relate to this campaign or SIRA's responsibilities. SIRA advised that these three additional statements were included to provide a broader context for any change in the green slip campaign survey results. For example, if all four measures reported an increase in positive responses of roughly the same size, then the increase may have been due to factors other than the advertising campaign.

This is not an appropriate use of the post-campaign evaluation, which should measure the success of the campaign against its stated objectives. The Guidelines list the purposes that government advertising may serve and none of these relate to improving the perception of the government. The inclusion of the above questions in SIRA's post-campaign evaluation creates a risk that the results may be used for party political purposes.

The campaign met most targets, however some were not challenging to achieve

The post-campaign evaluation demonstrated that the campaign met the targets for 12 of its 13 objectives including the targets relating to raising awareness of the refunds and the proportion of people claiming their refunds online. A fourteenth objective, the percentage of people aware that they should contact SIRA after a road accident injury, did not have a target set, meaning that it is not possible to say whether the campaign had the desired impact in this case.

In August 2017, before the campaign commenced, SIRA conducted a survey to determine the baselines for some of its objectives. This is a good practice to support an effective post campaign evaluation process. The survey found that 20 per cent of people were aware of the green slip reforms. SIRA's objective was to raise this to 25 per cent, which represents a small gain relative to the proposed campaign expenditure. The campaign aimed for 40 per cent of motorists to be aware of refunds, which is very low given that this was the primary focus of the campaign. SIRA followed the advice of its survey provider when setting these targets.

In the survey carried out after the campaign, 66 per cent of people were aware of the availability of green slip refunds for most motorists. The campaign also aimed to get 83 per cent of motorists to claim their refunds via online channels. It met this target, with a total of 84 per cent. Finally, 62 per cent of people in the post-campaign survey were aware of the green slip reforms. This result is discussed further below.

The overall target for total number of refunds claimed is 85 per cent of eligible drivers, that is to say CTP holders. SIRA will evaluate the results of this objective after the conclusion of the refund period in June 2019.

The campaign did little to inform the public about the broader green slip reforms

One objective of the green slip refund campaign was to inform the public about the green slip reforms. The final campaign creative material focused almost entirely on the green slip refunds rather than the range of other reforms. This was because the peer review raised concerns that the creative material was attempting to deliver too many messages.

The campaign submission stated that the advertising campaign would raise awareness of the broader reforms to the CTP scheme, citing several examples such as reduced opportunities for fraud and reduced insurer profits. SIRA also advised the Minister for Finance, Services and Property that secondary messaging in the campaign would benefit public understanding of the reforms.

Some of the television and radio advertisements referred to ‘more protection’ or ‘better protection’ for people injured on New South Wales roads, however advertisements did not refer to other elements of the reforms. Other campaign creative materials contained messages solely relating to the green slip refund and made no further reference to the broader reforms. SIRA used other communication channels, such as giving wallet cards to health service providers, to spread these messages to people, particularly those who had been injured.

Sixty two per cent of people in the post-campaign survey were aware of the green slip reforms. SIRA asked these people which benefits they associated with the reforms. The results of this survey are in Exhibit 4. Seventy-one per cent of this sample identified the reduced costs of green slips as one of the changes, but awareness of other elements of the reforms remains low. Though 29 per cent of people perceive the reforms to make the green slip scheme ‘fairer’, no more than 15 per cent of people could list a specific benefit which did not relate to insurance prices.

Exhibit 4: Perceived benefits associated with the changes to the CTP green slip scheme
Perceived benefit	Percentage aware of this benefit
Reduced costs of green slips for vehicle owners	71%
A fairer scheme for all people	29%
Reduced costs of comprehensive vehicle insurance	20%
Better support for people injured on our roads	15%
Less chances of fraudulent claims	15%
Lowering insurance company profits	13%
Quicker payment of claims to injured people	10%

Source: State Insurance Regulatory Authority.

Another campaign target was to ensure that people understood that they should contact SIRA in case of an injury. None of the campaign creative materials contained this information. SIRA did some limited work to inform the public about this through its social media channels. One of the pieces of creative material directed the reader to SIRA's website for further information on the reforms, which contained this information. During the campaign period, there was an increase in the number of calls received by SIRA's CTP Assist phone line. However, in the post-campaign evaluation, only two per cent of surveyed people identified that they should contact SIRA in case of an injury.

The media plan allowed sufficient time for cost-efficient media placement

During the peer review process, DPC provides advice to agencies about the time they should allow to ensure cost-efficient media placement. For example, DPC advise that agencies book television advertising six to 12 weeks in advance and that agencies book radio advertising two to eight weeks in advance.

SIRA allowed sufficient time between the completion of the peer review process and the commencement of the first advertising. SIRA signed the agreement with the approved Media Agency Services provider eight weeks before the campaign started, meaning that it could achieve cost-efficient media placement for all types of media used in this campaign.

SIRA directly negotiated with a single supplier, making it difficult to demonstrate value for money

SIRA directly negotiated with a single supplier to procure the campaign's creative material. A direct negotiation occurs when an agency negotiates with a proponent without first undergoing a competitive process. It is difficult to demonstrate value for money using direct negotiation due to the lack of competition.

ICAC's 'Guidelines for managing risks in direct negotiations' (ICAC Guidelines) provide guidance on how to undertake direct negotiations. SIRA has a direct negotiation checklist that aligns to the ICAC Guidelines. The SIRA checklist advises that staff should confirm that existing New South Wales prequalification schemes cannot provide the procurement before undertaking a direct negotiation. SIRA did not do this.

To procure creative materials, agencies can access the Advertising and Digital Communication Services prequalification scheme (the prequalification scheme). Using the prequalification scheme allows agencies to quickly seek quotes from suppliers who have a demonstrated track record and expertise. While agencies are not required to use the prequalification scheme, the NSW Procurement Board advises that agencies should use prequalification schemes where they are available to promote competition.

By using direct negotiation when the prequalification scheme was available, and by not seeking quotes from other suppliers, SIRA was acting in a way that reduced competition. This increases the risk that SIRA did not achieve value for money in its procurement of creative materials.

SIRA advised that it sought to ensure value for money by comparing the quote from its selected supplier with the amount spent on creative materials in other campaigns of similar size. SIRA did not document this analysis at the time or include it as part of the briefing note staff used to seek approval for undertaking direct negotiation. As a result, decision-makers were not fully informed when approving this engagement.

SIRA reported in a briefing note that it engaged in direct negotiations because:

it believed that the original timeframe did not allow for a competitive tender process
the supplier had done previous work on a related campaign for SIRA
the supplier provided sample work which received positive feedback from focus groups.

In July 2017, when peer review commenced, SIRA planned to launch the campaign in November 2017 to coincide with the beginning of the green slip reforms. SIRA believed that this timeframe was narrow enough to warrant entering direct negotiations. The ICAC Guidelines advise that a narrow timeframe is not a valid reason to enter into a direct negotiation. In late October 2017, the campaign launch was delayed until March 2018 to stagger the demand on the resources of Service NSW, which is administering the refund.

The ICAC Guidelines also advise against re-appointing a supplier because it has performed previous work. Instead, agencies could consider previous experience as one of several factors when deciding between quotes. In cases where an agency asks a supplier to provide sample work, the ICAC Guidelines advise that agencies should request sample work from multiple potential suppliers to promote competition.

The campaign's cost benefit analysis complied with the Act and Guidelines

The Act requires a cost benefit analysis (CBA) for any government advertising campaign likely to exceed $1.0 million in value. Section six of the Guidelines set out the requirements for a government advertising CBA. The campaign's CBA complied with the requirements of the Act and the Guidelines.

The campaign CBA could have demonstrated further cost effectiveness if it considered alternative media mixes as outlined in NSW Treasury's 'Cost Benefit Analysis Framework for Government Advertising and Information Campaigns'. This would also have been consistent with the Handbook.

The cluster Secretary signed the compliance certificate instead of the head of SIRA

The Act requires the head of the agency running the campaign to sign a compliance certificate.

The Secretary of the Department of Finance, Services and Innovation, the cluster to which SIRA belongs, signed the campaign's compliance certificate. However, section 17(2) of the State Insurance and Care Governance Act 2015 states that SIRA is ‘for the purposes of any Act, a NSW Government agency.’ Given this, the Chief Executive of SIRA was responsible for signing the compliance certificate for this campaign.

This is a minor non-compliance with the Act because the Chief Executive had reviewed the campaign and recommended that the Secretary sign the compliance certificate.

The NSW TAFE Commission (TAFE) ran the 'TAFE NSW 2018 Student Recruitment Annual Campaign Program' from November 2017 to September 2018. The aim of the campaign was to assist TAFE in achieving its 2018 student enrolment target by improving the perception of TAFE's brand and generating student enquiries. This is the first state-wide campaign run by TAFE operating under the One TAFE model. Previously, each TAFE Institute ran its own campaigns. The total budget of the campaign was $19.5 million. This audit examined only the semester one 2018 component of the campaign, which ran from November 2017 to April 2018 at a total cost of $9.5 million. See Appendix two for more details on this campaign.

The semester one component of the 'TAFE NSW 2018 Student Recruitment Annual Campaign Program' did not breach the specific provisions of section 6 of the Act which prohibits political advertising.

The campaign was not fully effective because it did not achieve its objective of reaching TAFE's semester one enrolment target.

The campaign was successful at achieving the campaign's targets which related to changing the public perception of TAFE.
TAFE was able to place most of its campaign media within cost-efficient timeframes. TAFE also achieved efficiencies by re-using many creative materials from a previous campaign.

TAFE could not demonstrate this campaign was carried out economically. TAFE's cost benefit analysis (CBA) for this campaign did not comply with three requirements of the Guidelines. For example, TAFE did not compare the campaign to a baseline case of not advertising.

The Guidelines require government advertising to be accurate in all statements. TAFE breached this requirement. The campaign material included one statement that was inaccurate and one that was overstated.

The revision of the Brand Guidelines in August 2017 impacted this campaign. TAFE re-used many creative materials that were created when TAFE was not required to include the NSW Government logo on its advertising material. DPC appears to have directed agencies that were launching advertising campaigns to immediately comply with the Brand Guidelines, however we could not find evidence that this advice was given to TAFE. As such, 59 per cent of TAFE's materials were not compliant with the Brand Guidelines at the launch of the campaign in November 2017. TAFE had made most of this campaign's creative materials compliant by June 2018.

The campaign materials we reviewed did not breach section 6 of the Act

Section 6 of the Act prohibits political advertising as part of a government advertising campaign. A government advertising campaign must not:

be designed to influence (directly or indirectly) support for a political party
contain the name, voice or image of a minister, a member of parliament or a candidate nominated for election to parliament
contain the name, logo, slogan or any other reference to a political party.

The audit team found no breaches of section 6 of the Act in the campaign material we reviewed.

The campaign achieved 16 of 24 objectives, but did not reach its enrolment target

The campaign had 24 objectives which had a target for semester one. TAFE set these targets using a combination of previous experience, corporate objectives and brand surveys.

The overall objective of the combined semester one and two campaigns was to support TAFE achieving its 2018 total enrolment target of 549,636. TAFE's semester one target was 361,350, which it did not achieve. This indicates that the campaign was not fully effective.

The campaign achieved 11 of its 16 output objectives. The output targets related to TAFE's media placements and ability to reach an audience efficiently. TAFE tracked progress against many of the campaign's output objectives daily. TAFE altered its media channels throughout the campaign meaning that some of the output objectives were not met because TAFE decided to focus on alternative media channels. The campaign also achieved all seven of its outcome objectives. The outcome objectives related to changing the public perception of TAFE.

TAFE's initial media plan allowed for efficient media placement

While TAFE's initial media plan allowed sufficient time between the approval of the campaign and its launch, a delay in receiving final approval for the campaign meant TAFE could not purchase media placements until two months later than planned. Most purchases still remained within DPC's recommended timeframes, but Indigenous television advertisements and metropolitan out of home advertisements both fell outside DPC's recommended time periods by one week. These delays did not impact on TAFE's efficiency.

TAFE re-used many creative materials, achieving some cost-savings

Rather than commissioning new creative materials, TAFE re-used many creative materials from the previous campaign and supplemented these with a selection of new creative materials. TAFE advised that this led to a cost saving of approximately $130,000.
TAFE sought quotes from suppliers on the government's Advertising and Digital Communication Services prequalification scheme for two creative material contracts. These contracts covered updates to existing materials and a selection of new materials.

The campaign's cost-benefit analysis did not comply with three requirements of the Guidelines

The Act requires an agency to conduct a cost-benefit analysis (CBA) if the cost of an advertising campaign is likely to exceed $1.0 million. The Guidelines set out the requirements of this CBA. TAFE did not comply with three of these requirements, outlined in Exhibit 5.

Exhibit 5: Guideline requirements for CBAs with which TAFE did not comply
6.2 The cost benefit analysis must isolate the additional costs and benefits attributable to the advertising campaign itself compared to the base-case of not-advertising. 6.3 The cost benefit analysis must specify the extent to which the expected benefits could be achieved without advertising. 6.4 The cost benefit analysis must outline what options other than advertising could be used to successfully implement the program and achieve the program benefits and a comparison of their costs.

Source: NSW Government Advertising Guidelines (2012).

In this circumstance, section 6.2 of the Guidelines required the CBA to identify the number of enrolments TAFE would expect if it did not advertise. TAFE advised us that it is not possible to say what this scenario would look like because there had always been some degree of advertising, however, this argument is not reflected in the CBA.

TAFE used 2017 as the baseline in the CBA. In 2017, TAFE spent $13.2 million on advertising. As such, the CBA was only able to isolate the impact of the increased expenditure rather than the impact of the campaign's entire $19.5 million expenditure. TAFE advised that 2017 had the most reliable state-wide data and this contributed to the decision to use it as the baseline.

During the audit, TAFE sought advice from NSW Treasury regarding whether a 2017 baseline was appropriate and NSW Treasury advised that it was. Regardless, TAFE did not receive this advice prior to writing the CBA and did not put commentary around this in the CBA. This would also not be sufficient for fulfilling the requirements of the Guidelines.

The CBA did not comply with sections 6.3 and 6.4 of the Guidelines. The CBA briefly considered the impact of spending the campaign budget directly on new training courses, however there was no sustained analysis of this option. TAFE staff advised that there are no realistic alternatives to advertising for achieving the campaign's objectives. However we did not see analysis to support this conclusion in documents provided to us.

The campaign CBA could have better demonstrated cost effectiveness if it considered alternative media mixes as outlined in NSW Treasury's 'Cost Benefit Analysis Framework for Government Advertising and Information Campaigns'. This would also have been consistent with the Handbook.

TAFE made one inaccurate claim in its advertising and overstated a second

The Guidelines set out rules regarding the content of a government advertising campaign. Exhibit 6 sets out one of the principles with which agencies must comply.

Exhibit 6: Guidelines' requirement for accuracy
The following principles apply to the style and content of government advertising campaigns: Accuracy in the presentation of all facts, statistics, comparisons and other arguments. All statements and claims of fact included in government advertising campaigns must be able to be substantiated.

Source: NSW Government Advertising Guidelines (2012).

TAFE made one inaccurate claim in its advertising and overstated a second.

In some campaign creative material, TAFE claimed that 78 per cent of its own graduates are employed after training (Exhibit 15 in Appendix 2). According to the National Centre for Vocational Education Research, 78 per cent of New South Wales Vocational Education and Training (VET) graduates (i.e. from all training providers) are employed after training. The result for TAFE graduates is 70.4 per cent.

One of the campaign's television advertisements refers to TAFE as ‘Australia's most reputable education provider’. This statement referred to a survey of current TAFE students who were asked where they would consider studying in future: TAFE, University or a private college. The current TAFE students selected TAFE by a large margin. The limited scope of TAFE's student survey and its results do not support the claim that it is ‘Australia's most reputable education provider’.

DPC did not consistently communicate the transitional arrangements for the Brand Guidelines and as such much of TAFE's creative material did not comply at campaign launch

On 7 August 2017, the government released the NSW Government Brand Guidelines (Brand Guidelines), setting out how agencies use the NSW Government logo. The Brand Guidelines replaced the Branding Style Guide which had been in place since September 2015. Some agencies were exempt from using the Branding Style Guide and the introduction of the new Brand Guidelines required these agencies to apply for a new exemption.

TAFE had recently commenced the peer review process for this campaign when the Brand Guidelines were released. TAFE was exempt from the requirements of the Branding Style Guide and as such the material which TAFE was planning to re-use in the new campaign did not contain the NSW Government logo.

Communication about how long agencies had to make themselves compliant with the Brand Guidelines was unclear. On 11 August 2017, the Chair of the Cabinet Standing Committee on Communication and Government Advertising (the Committee) sent a letter to the Secretary of the Department of Industry informing him that the Department must update all its material to be compliant with the Brand Guidelines ‘as soon as practicable within an 18-month transition period’. The Department of Industry advised TAFE that new advertising would need to be immediately compliant, however it was not clear if this included materials which agencies were re-using from previous campaigns. DPC advised the audit team that it expected re-used materials to be compliant when agencies launched new campaigns. DPC provided this advice to some agencies but did not communicate it more broadly. We could not source evidence that DPC provided this advice to TAFE.

DPC ran workshops to explain the transitional arrangements in September 2017 for the changes in the Brand Guidelines, however these did not specifically address the transitional timeframes for new advertising campaigns.

The Department of Industry, on behalf of TAFE, applied to the Committee for approval to co-brand the TAFE logo with the NSW Government logo. This was approved in October 2017. The requirements for co-branding are in Exhibit 7.

Exhibit 7: Co-branding in the NSW Government Brand Guidelines
Co-branding partners the agency logo with the NSW Government logo. The NSW Government logo must always be presented as the dominant or lead brand. The Brand Guidelines provide the following template shown below the exhibit box. The NSW Government logo is on the left and the agency logo is placed on the right, with a dividing line between them.

Appendix one - Responses from agencies

Appendix two - About the campaigns

Appendix three - About the audit

Appendix four - Performance auditing

Parliamentary reference - Report number #311 - released 18 December 2018

13 September 2018

Published

Progress and measurement of the Premier's Priorities

Premier and Cabinet

Compliance

Internal controls and governance

Management and administration

Project management

Risk

Service delivery

Shared services and collaboration

Workforce and capability

The Premier’s Implementation Unit uses a systematic approach to measuring and reporting progress towards the Premier’s Priorities performance targets, but public reporting needed to improve, according to a report released today by the Auditor-General of NSW, Margaret Crawford.

The Premier of New South Wales has established 12 Premier’s Priorities. These are key performance targets for government.

The 12 Premier's Priorities
150,000 new jobs by 2019	Reduce the volume of litter by 40 per cent by 2020
10 key projects in metro and regional areas to be delivered on time and on budget, and nearly 90 local infrastructure projects to be delivered on time	Increase the proportion of NSW students in the top two NAPLAN bands by eight per cent by 2019
Increase the proportion of women in senior leadership roles in the NSW Government sector from 33 to 50 per cent by 2025 and double the number of Aboriginal and Torres Strait Islander people in senior leadership roles in the NSW Government sector, from 57 to 114	Increase the proportion of young people who successfully move from Specialist Homelessness Services to long-term accommodation to more than 34 per cent by 2019
61,000 housing completions on average per year to 2021	Reduce the proportion of domestic violence perpetrators reoffending by 25 per cent by 2021
Improve customer satisfaction with key government services every year, this term of government to 2019	Decrease the percentage of children and young people re-reported at risk of significant harm by 15 per cent by 2020
81 per cent of patients through emergency departments within four hours by 2019	Reduce overweight and obesity rates of children by five percentage points by 2025

Source: Department of Premier and Cabinet, Premier’s Priorities website.

Each Premier’s Priority has a lead agency and minister responsible for achieving the performance target.

The Premier’s Implementation Unit (PIU) was established within the Department of Premier and Cabinet (DPC) in 2015. The PIU is a delivery unit that supports agencies to measure and monitor performance, make progress toward the Premier’s Priorities targets, and report progress to the Premier, key ministers and the public.

This audit assessed how effectively the NSW Government is progressing and reporting on the Premier's Priorities.

The Premier’s Implementation Unit (PIU) is effective in assisting agencies to make progress against the Premier’s Priorities targets. Progress reporting is regular but transparency to the public is weakened by the lack of information about specific measurement limitations and lack of clarity about the relationship of the targets to broader government objectives.The PIU promotes a systematic approach to measuring performance and reporting progress towards the Premier’s Priorities’ performance targets. Public reporting would be improved with additional information about the rationale for choosing specific targets to report on broader government objectives.

The PIU provides a systematic approach to measuring performance and reporting progress towards the Premier's Priorities performance targets. Public reporting would be improved with additional information about the rationale for choosing specific targets to report on broader government objectives. The data used to measure the Premier’s Priorities comes from a variety of government and external datasets, some of which have known limitations. These limitations are not revealed in public reporting, and only some are revealed in progress reported to the Premier and ministers. This limits the transparency of reporting.

The PIU assists agencies to avoid unintended outcomes that can arise from prioritising particular performance measures over other areas of activity. The PIU has adopted a collaborative approach to assisting agencies to analyse performance using data, and helping them work across organisational silos to achieve the Premier’s Priorities targets.

Data used to measure progress for some of the Premier’s Priorities has limitations which are not made clear when progress is reported. This reduces transparency about the reported progress. Public reporting would also be improved with additional information about the relationship between specific performance measures and broader government objectives.

The PIU is responsible for reporting progress to the Premier, key ministers and the public. Agencies provide performance data and some play a role in preparing progress reports for the Premier and ministers. For 11 of the Premier's Priorities, progress is reported against measurable and time-related performance targets. For the infrastructure priority, progress is reported against project milestones.

Progress of some Priorities is measured using data that has known limitations, which should be noted wherever progress is reported. For example, the data used to report on housing completions does not take housing demolitions into account, and is therefore overstating the contribution of this performance measure to housing supply. This known limitation is not explained in progress reports or on the public website.

Data used to measure progress is sourced from a mix of government and external datasets. Updated progress data for most Premier’s Priorities is published on the Premier’s Priorities website annually, although reported to the Premier and key ministers more frequently. The PIU reviews the data and validates it through fieldwork with front line agencies. The PIU also assists agencies to avoid unintended outcomes that can arise from prioritising single performance measures. Most, but not all, agencies use additional indicators to check for misuse of data or perverse outcomes.

We examined the reporting processes and controls for five of the Premier’s Priorities. We found that there is insufficient assurance over the accuracy of the data on housing approvals.

The relationships between performance measures and broader government objectives is not always clearly explained on the Premier’s Priority website, which is the key source of public information about the Premier’s Priorities. For example, the Premier’s Priority to reduce litter volumes is communicated as “Keeping our Environment Clean.” While the website explains why reducing litter is important, it does not clearly explain why that particular target has been chosen to measure progress in keeping the environment clean.

By December 2018, the Department of Premier and Cabinet should:

improve transparency of public reporting by:
- providing information about limitations of reported data and associated performance
- clarifying the relationship between the Premier’s Priorities performance targets and broader government objectives.
ensure that processes to check and verify data are in place for all agency data sources
encourage agencies to develop and implement additional supporting indicators for all Premier’s Priority performance measures to prevent and detect unintended consequences or misuse of data.

The Premier's Implementation Unit is effective in supporting agencies to deliver progress towards the Premier’s Priority targets.

The PIU promotes a systematic approach to monitoring and reporting progress against a target, based on a methodology used in delivery units elsewhere in the world. The PIU undertakes internal self-evaluation, and commissions regular reviews of methodology implementation from the consultancy that owns the methodology and helped to establish the PIU. However, the unit lacks periodic independent reviews of their overall effectiveness. The PIU has adopted a collaborative approach and assists agencies to analyse performance using data, and work across organisational silos to achieve the Premier’s Priorities targets.

Agency representatives recognise the benefits of being responsible for a Premier's Priority and speak of the value of being held to account and having the attention of the Premier and senior ministers.

By June 2019, the Department of Premier and Cabinet should:

establish routine collection of feedback about PIU performance including:
- independent assurance of PIU performance
- opportunity for agencies to provide confidential feedback.

Appendix one: Response from agency

Appendix two: Data sources

Appendix three: About the audit

Appendix four: Performance auditing

Parliamentary reference - Report number #307 - released 13 September 2018

10 August 2018

Published

Managing Antisocial behaviour in public housing

Community Services

Asset valuation

Infrastructure

Regulation

Service delivery

Workforce and capability

The Department of Family and Community Services (FACS) has not adequately supported or resourced its staff to manage antisocial behaviour in public housing according to a report released today by the Deputy Auditor-General for New South Wales, Ian Goodwin.

In recent decades, policy makers and legislators in Australian states and territories have developed and implemented initiatives to manage antisocial behaviour in public housing environments. All jurisdictions now have some form of legislation or policy to encourage public housing tenants to comply with rules and obligations of ‘good neighbourliness’. In November 2015, the NSW Parliament changed legislation to introduce a new approach to manage antisocial behaviour in public housing. This approach is commonly described as the ‘strikes’ approach.

When introduced in the NSW Parliament, the ‘strikes’ approach was described as a means to:

improve the behaviour of a minority of tenants engaging in antisocial behaviour
create better, safer communities for law abiding tenants, including those who are ageing and vulnerable.

FACS has a number of tasks as a landlord, including a responsibility to collect rent and organise housing maintenance. FACS also has a role to support tenants with complex needs and manage antisocial behaviour. These roles have some inherent tensions. The FACS antisocial behaviour management policy aims are:

to balance the responsibilities of tenants, the rights of their neighbours in social housing, private residents and the broader community with the need to support tenants to sustain their public housing tenancies.

This audit assessed the efficiency and effectiveness of the ‘strikes’ approach to managing antisocial behaviour in public housing environments.

We examined whether:

the approach is being implemented as intended and leading to improved safety and security in social housing environments
FACS and its partner agencies have the capability and capacity to implement the approach
there are effective mechanisms to monitor, report and progressively improve the approach.

Conclusion

FACS has not adequately supported or resourced its staff to implement the antisocial behaviour policy. FACS antisocial behaviour data is incomplete and unreliable. Accordingly, there is insufficient data to determine the nature and extent of the problem and whether the implementation of the policy is leading to improved safety and security.

FACS management of minor and moderate incidents of antisocial behaviour is poor. FACS has not dedicated sufficient training to equip frontline housing staff with the relevant skills to apply the antisocial behaviour management policy. At more than half of the housing offices we visited, staff had not been trained to:

conduct effective interviews to determine whether an antisocial behaviour complaint can be substantiated
de escalate conflict and manage complex behaviours when required
properly manage the safety of staff and tenants
establish information sharing arrangements with police
collect evidence that meets requirements at the NSW Civil and Administrative Tribunal
record and manage antisocial behaviour incidents using the information management system HOMES ASB.

When frontline housing staff are informed about serious and severe illegal antisocial behaviour incidents, they generally refer them to the FACS Legal Division. Staff in the Legal Division are trained and proficient in managing antisocial behaviour in compliance with the policy and therefore, the more serious incidents are managed effectively using HOMES ASB.

FACS provides housing services to most remote townships via outreach visits from the Dubbo office. In remote townships, the policy is not being fully implemented due to insufficient frontline housing staff. There is very limited knowledge of the policy in these areas and FACS data shows few recorded antisocial behaviour incidents in remote regions.

The FACS information management system (HOMES ASB) is poorly designed and has significant functional limitations that impede the ability of staff to record and manage antisocial behaviour. Staff at most of the housing offices we visited were unable to accurately record antisocial behaviour matters in HOMES ASB, making the data incorrect and unreliable.

Parliamentary reference - Report number #306 - released 10 August 2018

28 June 2018

Published

Regulation of water pollution in drinking water catchments and illegal disposal of solid waste

Environment

Compliance

Internal controls and governance

Management and administration

Regulation

Risk

There are important gaps in how the Environmental Protection Authority (EPA) implements its regulatory framework for water pollution in drinking water catchments and illegal solid waste disposal. This limits the effectiveness of its regulatory responses, according to a report released today by the Auditor-General for New South Wales, Margaret Crawford.

The NSW Environment Protection Authority (the EPA) is the State’s primary environmental regulator. The EPA regulates waste and water pollution under the Protection of the Environment Operations Act 1997 (the Act) through its licensing, monitoring, regulation and enforcement activities. The community should be able to rely on the effectiveness of this regulation to protect the environment and human health. The EPA has regulatory responsibility for more significant and specific activities which can potentially harm the environment.

Activities regulated by the EPA include manufacturing, chemical production, electricity generation, mining, waste management, livestock processing, mineral processing, sewerage treatment, and road construction. For these activities, the operator must have an EPA issued environment protection licence (licence). Licences have conditions attached which may limit the amount and concentrations of substances the activity may produce and discharge into the environment. Conditions also require the licensee to report on its licensed activities.

This audit assessed the effectiveness of the EPA’s regulatory response to water pollution in drinking water catchments and illegal solid waste disposal. The findings and recommendations of this review can be reasonably applied to the EPA’s other regulatory functions, as the areas we examined were indicative of how the EPA regulates all pollution types and incidents.

Conclusion

There are important gaps in how the EPA implements its regulatory framework for water pollution in drinking water catchments and illegal solid waste disposal which limit the effectiveness of its regulatory response. The EPA uses a risk-based regulatory framework that has elements consistent with the NSW Government Guidance for regulators to implement outcomes and risk-based regulation. However, the EPA did not demonstrate that it has established reliable practices to accurately and consistently detect the risk of non compliances by licensees, and apply consistent regulatory actions. This may expose the risk of harm to the environment and human health.

The EPA also could not demonstrate that it has effective governance and oversight of its regulatory operations. The EPA operates in a complex regulatory environment where its regional offices have broad discretions for how they operate. The EPA has not balanced this devolved structure with an effective governance approach that includes appropriate internal controls to monitor the consistency or quality of its regulatory activities. It also does not have an effective performance framework that sets relevant performance expectations and outcome-based key performance indicators (KPIs) for its regional offices.

These deficiencies mean that the EPA cannot be confident that it conducts compliance and enforcement activities consistently across the State and that licensees are complying with their licence conditions or the Act.

The EPA's reporting on environmental and regulatory outcomes is limited and most of the data it uses is self reported by industry. It has not set outcome-based key result areas to assess performance and trends over time.

The EPA uses a risk-based regulatory framework for water pollution and illegal solid waste disposal but there are important gaps in implementation that reduce its effectiveness.

Elements of the EPA’s risk-based regulatory framework for water pollution and illegal solid waste disposal are consistent with the NSW Government Guidance for regulators to implement outcomes and risk-based regulation. There are important gaps in how the EPA implements its risk-based approach that limit the effectiveness of its regulatory response. The EPA could not demonstrate that it effectively regulates licensees because it has not established reliable practices that accurately and consistently detect licence non compliances or breaches of the Act and enforce regulatory actions.

The EPA lacks effective governance arrangements to support its devolved regional structure. The EPA's performance framework has limited and inconclusive reporting on regional performance to the EPA’s Chief Executive Officer or to the EPA Board. The EPA cannot assure that it is conducting its regulatory responsibilities effectively and efficiently.

The EPA does not consistently evaluate its regulatory approach to ensure it is effective and efficient. For example, there are no set requirements for how EPA officers conduct mandatory site inspections, which means that there is a risk that officers are not detecting all breaches or non-compliances. The inconsistent approach also means that the EPA cannot rely on the data it collects from these site inspections to understand whether its regulatory response is effective and efficient. In addition, where the EPA identifies instances of non compliance or breaches, it does not apply all available regulatory actions to encourage compliance.

The EPA also does not have a systematic approach to validate self-reported information in licensees’ annual returns, despite the data being used to assess administrative fees payable to the EPA and its regulatory response to non-compliances.

The EPA does not use performance frameworks to monitor the consistency or quality of work conducted across the State. The EPA has also failed to provide effective guidance for its staff. Many of its policies and procedures are out-dated, inconsistent, hard to access, or not mandated.

Recommendations

By 31 December 2018, to improve governance and oversight, the EPA should:
1.	implement a more effective performance framework with regular reports to the Chief Executive Officer and to the EPA Board on outcomes-based key result areas that assess its environmental and regulatory performance and trends over time
By 30 June 2019, to improve consistency in its practices, the EPA should:
2.	progressively update and make accessible its policies and procedures for regulatory operations, and mandate procedures where necessary to ensure consistent application
3.	implement internal controls to monitor the consistency and quality of its regulatory operations.

The EPA does not apply a consistent approach to setting licence conditions for discharges to water.

The requirements for setting licence conditions for water pollution are complex and require technical and scientific expertise. In August 2016, the EPA approved guidance developed by its technical experts in the Water Technical Advisory Unit to assist its regional staff. However, the EPA did not mandate the use of the guidance until mid-April 2018. Up until then, the EPA had left discretion to regional offices to decide what guidance their staff use. This meant that practices have differed across the organisation. The EPA is yet to conduct training for staff to ensure they consistently apply the 2016 guidance.

The EPA has not implemented any appropriate internal controls or quality assurance process to monitor the consistency or quality of licence conditions set by its officers across the State. This is not consistent with good regulatory practice.

The triennial 2016 audit of the Sydney drinking water catchment report highlighted that Lake Burragorang has experienced worsening water quality over the past 20 years from increased salinity levels. The salinity levels were nearly twice as high as in other storages in the Sydney drinking water catchment. The report recommended that the source and implication of the increased salinity levels be investigated. The report did not propose which public authority should carry out such an investigation.

To date, no NSW Government agency has addressed the report's recommendation. There are three public authorities, the EPA, DPE and WaterNSW that are responsible for regulating activities that impact on water quality in the Sydney drinking water catchment, which includes Lake Burragorang.

Recommendation

By 30 June 2019, to address worsening water quality in Lake Burragorang, the EPA should:
4.	(a) review the impact of its licensed activities on water quality in Lake Burragorang, and
	(b) develop strategies relating to its licensed activities (in consultation with other relevant NSW Government agencies) to improve and maintain the lake's water quality.

The EPA’s risk-based approach to monitoring compliance of licensees has limited effectiveness.

The EPA tailors its compliance monitoring approach based on the performance of licensees. This means that licensees that perform better have a lower administrative fee and fewer mandatory site inspections.

However, this approach relies on information that is not complete or accurate. Sources of information include licensees’ annual returns, EPA site inspections and compliance audits, and pollution reports from the public.

Licensees report annually to the EPA on their performance, including compliance against their licence conditions. The Act contains significant financial penalties if licensees provide false and misleading information in their annual returns. However, the EPA does not systematically or consistently validate information self-reported by licensees, or consistently apply regulatory actions if it discovers non-compliance.

Self-reported compliance data is used in part to assess a licensed premises’ overall environmental risk level, which underpins the calculation of the administrative fee, the EPA’s site inspection frequency, and the licensee’s exposure to regulatory actions. It is also used to assess the load-based licence fee that the licensee pays.

The EPA has set minimum mandatory site inspection frequencies for licensed premises based on its assessed overall risk level. This is a key tool to detect non-compliance or breaches of the Act. However, the EPA has not issued a policy or procedures that define what these mandatory inspections should cover and how they are to be conducted. We found variations in how the EPA officers in the offices we visited conducted these inspections. The inconsistent approach means that the EPA does not have complete and accurate information of licensees’ compliance. The inconsistent approach also means that the EPA is not effectively identifying all non-compliances for it to consider applying appropriate regulatory actions.

The EPA also receives reports of pollution incidents from the public that may indicate non-compliance. However, the EPA has not set expected time frames within which it expects its officers to investigate pollution incidents. The EPA regional offices decide what to investigate and timeframes. The EPA does not measure regional performance regarding timeframes.

The few compliance audits the EPA conducts annually are effective in identifying licence non-compliances and breaches of the Act. However, the EPA does not have a policy or required procedures for its regulatory officers to consistently apply appropriate regulatory actions in response to compliance audit findings.

The EPA has not implemented any effective internal controls or quality assurance process to check the consistency or quality of how its regulatory officers monitor compliance across the State. This is not consistent with good regulatory practice.

Recommendations

To improve compliance monitoring, the EPA should implement procedures to:
5.	by 30 June 2019, validate self-reported information, eliminate hardcopy submissions and require licensees to report on their breaches of the Act and associated regulations in their annual returns
6.	by 31 December 2018, conduct mandatory site inspections under the risk-based licensing scheme to assess compliance with all regulatory requirements and licence conditions.

The EPA cannot assure that its regulatory enforcement approach is fully effective.

The EPA’s compliance policy and prosecution guidelines have a large number of available regulatory actions and factors which should be taken into account when selecting an appropriate regulatory response. The extensive legislation determining the EPA’s regulatory activities, and the devolved regional structure the EPA has adopted in delivering its compliance and regulatory functions, increases the risk of inconsistent compliance decisions and regulatory responses. A good regulatory framework needs a consistent approach to enforcement to incentivise compliance.

The EPA has not balanced this devolved regional structure with appropriate governance arrangements to give it assurance that its regulatory officers apply a consistent approach to enforcement.

The EPA has not issued standard procedures to ensure consistent non-court enforcement action for breaches of the Act or non-compliance with licence conditions. Given our finding that the EPA does not effectively detect breaches and non-compliances, there is a risk that it is not applying appropriate regulatory actions for many breaches and non-compliances.

A recent EPA compliance audit identified significant non-compliances with incident management plan requirements. However, the EPA has not applied regulatory actions for making false statements on annual returns for those licensees that certified their plans complied with such requirements. The EPA also has not applied available regulatory actions for the non-compliances which led to the false or misleading statements.

Recommendation

By 31 December 2018 to improve enforcement, the EPA should:
7.	Implement procedures to systematically assess non-compliances with licence conditions and breaches of the Act and to implement appropriate and consistent regulatory actions.

The EPA has implemented the actions listed in the NSW Illegal Dumping Strategy 2014–16. To date, the EPA has also implemented four of the six recommendations made by the ICAC on EPA's oversight of Regional Illegal Dumping Squads.

The EPA did not achieve the NSW Illegal Dumping Strategy 2014–16 target of a 30 per cent reduction in instances of large scale illegal dumping in Sydney, the Illawarra, Hunter and Central Coast from 2011 levels.

In the reporting period, the incidences of large scale illegal dumping more than doubled. The EPA advised that this increase may be the result of greater public awareness and reporting rather than increased illegal dumping activity.

By June 2018, the EPA is due to implement one outstanding recommendation made by the ICAC but has not set a time for the other outstanding recommendation.

Appendix one – Response from agency

Appendix two – List of enforcement tools

Appendix three – The EPA's organisational structure

Appendix four – The EPA's regions and branches

Appendix five – About the audit

Appendix six – Performance auditing

Parliamentary reference - Report number #304 - released 28 June 2018