Government advertising 2021–22

Report highlights

What the report is about

The Government Advertising Act 2011 requires the Auditor-General to undertake a performance audit on government advertising activities each financial year.

This audit examined whether TAFE NSW's annual advertising campaign in 2021–22:

  1. was carried out effectively, economically, and efficiently
  2. complied with regulatory requirements and the Government Advertising Guidelines.

What we found

TAFE NSW complied with Section 6 of the Act, prohibiting political content.

It also complied with most other advertising requirements.
 
An important exception was that the Managing Director certified that the campaign complied with regulatory requirements and was an efficient and cost-effective means of achieving its public purpose, before a cost-benefit analysis (CBA) was completed.

We have found issues with agencies complying with CBA requirements in previous government advertising audits. This includes the failure to complete them before signing compliance certificates.

The policy owner, the Department of Customer Service (DCS), does not consider oversight of CBAs to be within the scope of their peer review process.  

TAFE NSW evaluated this advertising campaign by surveying a population significantly broader than the target audience. As such, survey results may not accurately reflect the views of the intended audience.

What we recommended

By 30 June 2023, TAFE NSW should:

  1. implement processes that ensure:
    1. CBAs are completed before the launch of campaigns over $1 million
    2. compliance certificates are completed only after all regulatory requirements are met
  2. consider adding to its current evaluation methods by surveying a population which closely reflects the age profile of its intended target audience.

By June 2023, DCS should:

  1. improve whole‑of‑government reporting and monitoring processes to provide the NSW Government with a central view of compliance, including the completion of CBAs by agencies.

Fast facts

Executive summary

The Government Advertising Act 2011 (the Act) sets out requirements that must be followed by a government agency when it carries out a government advertising campaign. The requirements include an explicit prohibition on political advertising, as well as a need to complete a peer review and cost-benefit analysis before the campaign commences. The accompanying Government Advertising Regulation 2018 (the Regulation) and Government Advertising Guidelines (the Guidelines) address further matters of detail.

The Act also requires the Auditor-General to conduct a performance audit on the activities of one or more government agencies in relation to government advertising campaigns in each financial year. The performance audit must assess whether a government agency (or agencies) has carried out activities in relation to government advertising campaigns in an effective, economical and efficient manner. It also assesses compliance with the Act, the Regulation, other laws and the Guidelines.

This audit examined TAFE NSW's advertising campaign for the 2021–22 financial year. TAFE NSW is the NSW Government's public provider of vocational education and training. TAFE NSW carries out an advertising campaign every year. In 2021–22, it spent $15.16 million on developing and implementing advertising. TAFE NSW used channels such as television, radio, internet and social media, press, and out of home advertising in public settings such as bus stops. The advertising aimed to increase the percentage of people considering TAFE NSW for training or education, grow the percentage of people who consider TAFE NSW to be the preferred education provider in NSW, and maintain the proportion of people who are aware of TAFE NSW more generally.

There are a range of private service providers helping to deliver vocational education and training in NSW.

Conclusion

TAFE NSW’s advertising campaign for 2021–22 was for an allowed purpose under the Act and did not include political advertising. TAFE NSW complied with most of the requirements set out in the Act, the Regulation, and the Guidelines, but it failed to complete a cost-benefit analysis for the campaign or provide sufficient support for the compliance certificate signed by TAFE NSW's Managing Director.

TAFE NSW complied with the requirement to complete a peer review of its campaign, but it did not meet the requirement to complete a cost-benefit analysis, either before it launched the campaign or during its implementation throughout 2021–22. Some of TAFE NSW's advertising did not meet the requirement for statements to be clearly supported by evidence.

The Act requires the head of an agency to sign a compliance certificate stating that, among other things, the campaign complies with the Act, the Regulation, and the Guidelines, and that the campaign is an efficient and cost-effective means of achieving the public purpose. TAFE NSW's Managing Director signed a compliance certificate in May 2021. However, TAFE NSW had not prepared a cost-benefit analysis as required under the Act and therefore TAFE NSW's Managing Director could not validly sign the compliance certificate. TAFE NSW did not subsequently complete a cost-benefit analysis during the campaign.

The campaign achieved many of its objectives and other performance measures and is likely to have been impactful. It is also likely that TAFE NSW’s advertising campaign in 2021–22 represented economical, efficient, and effective spend. However, the lack of a cost-benefit analysis meant that this could not be confidently demonstrated by TAFE NSW.

TAFE NSW used internal resources to create its advertising content, such as videos, radio scripts and press advertising, and relied upon a specialist partner to arrange and place its media in the appropriate advertising channel. TAFE NSW also adjusted the advertising campaign in response to performance data and in response to changes in the educational and advertising marketplaces.

TAFE NSW evaluated the impact of its advertising and tracked its brand performance using a survey which reflected the New South Wales general population aged between 16 and 60. However, this evaluation did not match TAFE NSW's advertising spend as TAFE NSW directed significantly more of its campaign budget to influencing younger people in this cohort.

1. Key findings

Campaign materials we reviewed did not contain political content and were for an allowable purpose

The audit team reviewed campaign materials developed as part of TAFE NSW’s paid advertising campaign including radio transcripts, digital videos, and display. The audit found no breaches of Section 6 of the Act, which prohibits advertising as part of a government advertising campaign.

TAFE NSW's advertising was also consistent with Section 4 of the Act and Section 1.2 of the NSW Government Advertising Guidelines, which define the nature of, and set out the allowable purposes for, government advertising.

TAFE NSW completed a peer review for its annual advertising campaign, but it did not complete a cost-benefit analysis

Section 7 of the Act and Section 4 of the Regulation require the head of a government agency to ensure, before the start of the campaign, that:

  • a peer review of the proposed campaign is carried out if the cost of the campaign is likely to exceed $250,000
  • a cost-benefit analysis is carried out if the cost of the campaign is likely to exceed $1 million.

The peer review is an independent review of the need for the advertising campaign, the creative and media strategy and how the agency will manage the campaign. The Department of Customer Service (DCS) manages the peer review process. In May 2021, DCS confirmed that TAFE NSW had met the peer review requirements and advised TAFE NSW that a cost-benefit analysis was required as the planned campaign budget exceeded the $1 million threshold. However, TAFE NSW did not complete a cost-benefit analysis before or during its annual advertising campaign for 2021–22. The absence of a cost-benefit analysis meant that TAFE NSW did not comply with the Act.

TAFE NSW's Managing Director signed an Advertising Compliance Certificate without full supporting evidence

Section 8 of the Act requires the head of an agency to certify that the planned campaign complies with the Act, the Regulation, and the Guidelines.

In May 2021, TAFE NSW’s Managing Director signed a compliance certificate for the campaign. The certificate was accompanied by a submission which highlighted key metrics from an evaluation of the previous year’s advertising. However, the submission did not include the required cost-benefit analysis and did not meet the requirement in the Guidelines to consider options other than advertising to achieve the campaign's objectives. As a result, the Managing Director could not validly certify that the campaign complied with the Act, the Regulation, and the Guidelines.

DCS recommended this campaign proceed without sighting key documentation required under the NSW Government's regulatory framework

DCS is the central government agency responsible for overseeing government advertising. It prepares the annual consolidated submission of all advertising proposals for consideration by the Expenditure Review Committee (ERC) of NSW Cabinet. DCS recommended this advertising campaign without sighting a cost-benefit analysis and without requiring TAFE NSW to subsequently forward to DCS a copy of a completed cost-benefit analysis.

DCS considers the completion of a cost-benefit analysis to be an agency responsibility and does not follow up to ensure they are completed. Previous audits have found that some agencies had not satisfactorily completed cost-benefit analyses or other documentation required under the Government's regulatory framework. This lack of oversight on key documentation required under the regulatory framework (particularly cost-benefit analyses) reduces the ability of the Government to be confident that advertising expenditure is:

  • consistent with regulatory requirements
  • an efficient, effective, and economical use of funds.

TAFE NSW’s advertising did not always comply with requirements to present information accurately and objectively

The Guidelines require that all facts, statistics, comparisons, and other arguments are presented accurately, and all statements and claims of fact in Government advertising must be able to be substantiated.

TAFE NSW operates in a competitive environment and needs to market its educational services to current and prospective students. However, to meet the Government's advertising guidelines, TAFE NSW's advertising must also be evidence-based. The audit considers that TAFE NSW's advertising did not meet this need when it claimed to be Australia's leading training provider, as opposed to one of Australia's leading training providers or Australia’s largest training provider.

TAFE NSW established relevant objectives and measures for the campaign, achieving slightly over half of its targets in outputs and outcomes, and all its impact targets

TAFE NSW complied with the NSW Government evaluation framework. It prepared a one-page summary of the campaign’s objectives and its links to organisational and government objectives, the target audience and key messages, and the planned inputs, activities, outcomes and impacts. It also prepared a ‘Campaign Objectives Table’ that sets out detailed information on baseline and targeted performance, measurement milestones and evaluation methods.

The campaign met 16 of 32 output targets, seven of ten outcome targets and all four of the impact targets. The campaign only narrowly failed to achieve some of its intended targets. In some instances, TAFE NSW’s performance against these targets was affected by events such as lockdowns related to the COVID-19 pandemic, which led to the closure of cinemas and some newspapers, which had an impact on TAFE NSW’s intended levels of cinema and print advertising.

The absence of a cost-benefit analysis for this campaign means the decision to invest in this campaign was made without a clear view of potential value for money

TAFE NSW’s approved advertising campaign budget was $15.25 million in 2021–22, which significantly exceeded the legislative threshold of $1 million that requires a cost-benefit analysis to be carried out before the start of the campaign. TAFE NSW prepared a rationale to support its advertising investment but did not prepare the required cost-benefit analysis. A cost-benefit analysis was also not completed for the prior year's campaign. This limited the information that the Government had available to it when it decided to invest in the campaign.

TAFE NSW evaluated performance by surveying a population that was significantly broader than its target audience

TAFE NSW evaluated the effect of its advertising using a twice-yearly survey of consumer preferences and views. The sample population for this survey included:

  • 20% of respondents aged 16 to 24, despite TAFE NSW directing 45% of its planned spending at people in that age group
  • 33% of respondents aged 45 to 60, despite TAFE NSW targeting 10% of its planned advertising spend at this age group.

TAFE NSW states that it tracks the performance of its advertising by surveying a population that reflects the age profile of New South Wales residents aged between 16 and 60. TAFE NSW considers that this approach provides insight into its brand performance more generally. It further states that the survey's proportion of people over 40 is appropriate as these people will frequently influence education decisions made by younger people.

During this audit, TAFE NSW re-calculated its survey results from June 2022 using an age profile that reflects its current student population, which is significantly younger than New South Wales general population and closer to TAFE NSW's planned spend by audience age. TAFE NSW stated that the recalculated survey found the proportion of people who would consider TAFE NSW for future study was reduced from 69% to 65%.

TAFE NSW adjusted the delivery of its annual advertising campaign throughout the year in response to business data, which assisted with more efficient expenditure

TAFE NSW prepared an annual advertising plan which was reviewed by DCS to ensure the planned advertising had evidence of its likely effectiveness, appropriately used government communication channels, and was informed by customer and behavioural insights. TAFE NSW then worked with its media partner to execute the advertising plan by placing media throughout the year.

TAFE NSW also adjusted its investment in various advertising channels throughout the year in response to changes in:

  • business performance measures, such as lower-than-expected student enrolments in specific localities
  • operational performance measures, such as when digital advertising resulted in too many users clicking through to a website but then rapidly leaving it, known as a high ‘bounce factor’.

The combination of a peer reviewed overall annual advertising plan and appropriate adjustments to events and data throughout the year means that TAFE NSW’s spend on advertising is likely to be efficient.

TAFE NSW used in-house capability to develop the creative content, and whole-of-government arrangements to ensure it paid market rates, which improving the overall economy of the campaign

TAFE NSW spent just over 79% of its total advertising spend in 2021–22 to buy advertising media, such as social media advertisements, out of home advertising, and print advertising. It used its media partner to arrange and place this media, using prices which had been negotiated by another communications agency on behalf of all NSW Government. TAFE NSW developed the creative content for the campaign, with still photography and video shot on campus using staff and students in normal contexts.

The advertising rates paid by TAFE NSW under the whole-of-government contract were reviewed annually throughout the life of the contract and were independently market tested in 2018–19. TAFE NSW also states that some of its advertising spend was at prices significantly lower than the whole-of-government contract. Consequently, the prices paid by TAFE NSW for advertising in 2021–22 were likely to have been market rates.

2. Recommendations

By 30 June 2023, TAFE NSW should:

1. implement processes that ensure:
a) cost-benefit analyses are completed before the launch of a government advertising campaign over $1 million
b) advertising compliance certificates are completed by the Managing Director only after all regulatory requirements have been met

2. consider adding to its current evaluation methods by surveying a population which closely reflects the age profile of its intended target audience.

By 30 June 2023, the Department of Customer Service should:

3. improve whole-of-government reporting and monitoring processes to provide the NSW Government with a central view of compliance across all aspects of the government advertising regulatory framework, including the completion of agency cost-benefit analyses.

1. Introduction

1.1 Background

What is government advertising?

Governments use advertising to communicate information about a government program, policy or initiative to members of the public. Government advertising is funded by or on behalf of a government agency and is disseminated under a commercial advertising distribution agreement through a variety of media, such as radio, television, the internet, newspapers, billboards or cinemas.

There are three broad categories of government advertising:

  • Recruitment advertising – advertising which promotes specific job vacancies and employment opportunities within a government agency.
  • Public notices – advertising which communicates a clear, simple message or announcement and is generally one-off or short-term in nature. 
  • Public awareness advertising – coordinated communications to raise awareness of key issues, such as government initiatives, or to encourage behaviour change.

How much is spent on government advertising in New South Wales?

The NSW Government spent $135.46 million on advertising in 2021–22. The overall expenditure across previous financial years is provided in Exhibit 1.

Exhibit 1: NSW Government media expenditure from 2015-16 to 2021-22	The chart plots overall expenditure on government advertising across government from 2015-16 to 2021-22.
Exhibit 1: NSW Government media expenditure from 2015–16 to 2021–22
Source: Audit Office analysis.

How is government advertising regulated?

There is a regulatory framework that includes both policy and legislation governing NSW Government advertising. Exhibit 2 contains an overview of this regulatory framework.

Exhibit 2: Overview of the government advertising regulatory framework
Element Purpose
Government Advertising Act 2011 (the Act) Sets out legal requirements for government advertising.
Government Advertising Regulation 2018 (the Regulation) Sets out a threshold for the requirement to complete a peer
review and defines exemptions to the Act.
NSW Government Advertising Guidelines 2012 (the Guidelines) Sets out requirements in relation to the style and content, dissemination and cost of government advertising campaigns, as well as guidance on cost-benefit analyses and peer reviews.
Source: Audit Office analysis.

DCS has policy responsibility for government advertising. The heads of each government agency are responsible for their own agency's compliance with the regulatory framework.

Prohibition of political advertising

Section 6 of the Act prohibits political advertising as part of a government advertising campaign. Government advertising campaigns must not:

  • be designed to influence (directly or indirectly) support for a political party
  • contain the name, voice or image of a minister, a member of Parliament or a candidate nominated for election to Parliament
  • contain the name, logo, slogan or any other reference to a political party.

In addition, the Guidelines require government advertising campaigns to be politically neutral and clearly distinguishable from party political messages.

Requirements prior to the commencement of a campaign

The Act states that a government advertising campaign must not commence unless the head of the agency has signed a compliance certificate for the campaign. This compliance certificate states that the head of the agency believes the government advertising campaign:

  • complies with the Act, Regulation and Guidelines
  • contains accurate information
  • is necessary to achieve a public purpose and is supported by analysis and research
  • is an efficient and cost-effective means of achieving its public purpose.

The Act defines further requirements for campaigns which are likely to exceed a total cost of $250,000 or $1 million. 

Exhibit 3 summarises the key legislative and policy requirements that agencies must meet before commencing an advertising campaign.

Exhibit 3: Requirements before an advertising campaign can commence
Advertising cost Compliance certificate Budget approval by Cabinet committee Campaign review Cost-benefit analysis
<$250,000
-- -- --
$250,000 to $1 million
--
>$1 million
Source: Audit Office analysis. 

Government advertising campaigns likely to cost over $250,000 are subject to peer review by DCS before the campaign commences. Peer review involves DCS employees assessing the proposed advertising campaign against criteria outlined in the Guidelines.

Peer review is a two-stage process:

  1. Budget approval – government agencies submit an advertising budget proposal to the DCS Campaign Effectiveness Team. The Campaign Effectiveness Team reviews the budget proposal and makes recommendations to the Expenditure Review Committee (ERC) of Cabinet. ERC is responsible for approving campaign budgets.
  2. Campaign review – the DCS Campaign Effectiveness Team assesses the need for the proposed advertising campaign, the creative and media strategy (including objectives and target audiences) and how the agency will manage the campaign. Specifically, the campaign review considers:
    • alignment with government priorities and commitments, and the life of NSW customers
    • effectiveness of approach to regional customers, Aboriginal customers and Culturally and Linguistically Diverse (CALD) customers
    • creative strategy and its alignment to campaign objectives
    • media strategy and its alignment to campaign objectives, including effectiveness of consideration of digital channels
    • how the advertising integrates with other communication approaches
    • clarity and appropriateness of output, outcome and impact objectives, measures and targets, and proposed campaign evaluation approaches
    • likely effectiveness of the campaign based on previous campaign performance (if repeat) and campaign research.

Peer reviews are an important step in a campaign as they provide independent assurance for the agency regarding the campaign's compliance with some legislative and policy obligations. Peer reviews assess that government advertising campaigns are necessary, are likely to be effective, and that they are established with suitable objectives, measures and targets.

The Act further requires an agency proposing to run a government advertising campaign that is likely to cost over $1 million to complete a cost-benefit analysis before the campaign commences. A cost-benefit analysis aims to evaluate the net economic cost or benefit of the campaign and indicate to decision-makers how the campaign will affect the wellbeing of NSW residents. It can also play an important role as a baseline for post-campaign evaluation. 

1.2 About TAFE NSW

TAFE NSW is the NSW Government's public provider of vocational education and training. TAFE NSW is focused on building a strong economy through a skilled and agile workforce. It is Australia’s largest training provider with over 411,500 course enrolments in 2021–22.

TAFE NSW employed 5,730 teachers and 4,592 support staff and had operating expenses of $1.88 billion in 2021–22.

1.3 About the audited campaign

TAFE NSW carries out an advertising campaign every year. In 2021–22, the advertising aimed to increase the percentage of people considering TAFE NSW for training or education, grow the percentage of people who consider TAFE to be the preferred education provider in NSW and, more generally, maintain awareness of TAFE NSW.

The campaign was primarily focused on potential and current students located in New South Wales in the following age groups:

  • 15 to 19 (defined by TAFE NSW as pathfinders)
  • 20 to 24 (career starters)
  • 25 to 44 (career progressors)
  • 45 years and over (career changers).

The campaign also targeted decision-makers in small and medium-sized businesses, as well as prospective international students. It occurred throughout the year and included advertising activities on television, radio, internet and social media, press, and out of home (or billboard) advertising in transport and other settings.

TAFE's advertising campaign expenditure was $15,157,918. Campaign delivery was impacted by COVID-19 restrictions (particularly during the second half of 2021) and flooding in some TAFE locations.

Appendix two contains further details about the 2021–22 campaign.

1.4 About this audit

The Act requires the Auditor-General of New South Wales to conduct a performance audit on the activities of one or more government agencies in relation to government advertising campaigns in each financial year.

In conducting the audit, the Auditor-General must determine whether a government agency (or agencies) has carried out activities in relation to government advertising in an effective, economical and efficient manner, and in compliance with the Act, the Guidelines and other laws. Appendix three contains further details about this audit.

The audit team did not review the broader use of TAFE NSW's social media outside paid social media content relevant to these campaigns. Unpaid social media content is not considered government advertising for the purposes of the Act.

2. Compliance with the regulatory framework

This part of the report sets out key aspects of TAFE NSW's compliance with the government advertising regulatory framework. It considers whether TAFE NSW complied with the:

  • Government Advertising Act 2011
  • Government Advertising Regulation 2018
  • NSW Government Advertising Guidelines 2012 and other relevant policy.

2.1 Compliance with the Act and Regulation

Campaign materials we reviewed did not contain political content

Section 6 of the Act prohibits political advertising as part of a government advertising campaign. A government advertising campaign must not:

  • be designed to influence (directly or indirectly) support for a political party
  • contain the name, voice or image of a minister, a member of Parliament or a candidate nominated for election to Parliament
  • contain the name, logo, slogan or any other reference to a political party.

The audit team reviewed campaign materials developed as part TAFE NSW’s paid advertising campaign including radio transcripts, digital videos, and display. The audit found no breaches of Section 6 of the Act in the campaign material reviewed.

All reviewed campaign materials were for an allowable purpose

Section 4 of the Act states that government advertising campaigns are ‘the dissemination to members of the public of information about a government program, policy or initiative, or about any public health or safety or other matter’. To support this, Section 1.2 of the NSW Government Advertising Guidelines states that government advertising campaigns may only be used to achieve certain objectives, two of which are to encourage:

  • changed behaviours or attitudes that will lead to improved public health and safety or quality of life
  • use of government products and services.

The audit team considers that TAFE NSW’s advertising campaign was consistent with these objectives.

TAFE NSW completed a peer review for its annual advertising campaign, but did not complete a cost-benefit analysis

Section 7 of the Act and Section 4 of the Regulation require the head of a government agency to ensure, before the start of the campaign, that:

  • a peer review of the proposed campaign is carried out if the cost of the campaign is likely to exceed $250,000
  • a cost-benefit analysis is carried out if the cost of the campaign is likely to exceed $1 million.

Section 7(4) of the Act contains a provision that allows for cost-benefit analysis and peer review to be undertaken after the start of a campaign if the head of the agency ‘is satisfied that the campaign relates to an urgent public health or safety matter or is required in other urgent circumstances’. As a recurring campaign, TAFE NSW annual advertising does not generally fit this criterion.

The peer review is an independent review of the need for the proposed advertising campaign, the creative and media strategy (including objectives and target audiences) and how the agency will manage the campaign. The Department of Customer Service (DCS) manages the peer review process.

On 24 May 2021, DCS wrote to TAFE NSW to confirm that it had met the peer review requirements and advised TAFE NSW that, as the campaign budget exceeds $1 million, a cost-benefit analysis was required prior to campaign commencement. However, TAFE NSW did not complete a cost-benefit analysis before or during its annual advertising campaign for 2021–22. A cost-benefit analysis was also not completed for the 2020–21 annual advertising campaign. The absence of a cost-benefit analysis meant that TAFE NSW did not comply with the Act.

TAFE NSW's Managing Director signed an Advertising Compliance Certificate without full supporting evidence

Under the Act, the head of an agency must sign a compliance certificate prior to commencement of a campaign. The compliance certificate is required by Section 8 of the Act and states that the head of the agency confirms that a proposed government advertising campaign:

  • complies with the Act, the Regulation, and the Guidelines
  • contains accurate information
  • is necessary to achieve a public purpose and is supported by analysis and research
  • is an efficient and cost-effective means of achieving that public purpose.

On 28 May 2021, TAFE NSW’s Managing Director signed a compliance certificate for the campaign. The certificate was accompanied by a submission that highlighted key metrics from an evaluation of the previous year’s advertising. This showed that business and individuals were more likely than in previous years to consider TAFE for training or education, as well as improvements to metrics such as the number of website sessions, course enquiries and course enrolments. These metrics provide some support for the certification that the campaign was ‘an efficient and cost-effective means of achieving the purpose.’ However, they do not satisfy either the:

  • Government Advertising Act 2011 requirement for agencies to conduct a cost-benefit analysis for any campaign likely to exceed $1 million
  • Government Advertising Guidelines' requirement for agencies to consider options other than advertising to achieve the desired objective.

As a result, the Managing Director could not validly certify that the campaign complied with the Act, the Regulation, and the Guidelines.

DCS recommended this campaign proceed without sighting key documentation required under the Government's regulatory framework

DCS prepares the annual consolidated submission of all advertising proposals for consideration by the ERC, and is the central government agency responsible for overseeing government advertising. DCS completed the peer review of TAFE NSW’s advertising campaign in May 2021 without sighting a cost-benefit analysis and without requiring TAFE NSW to subsequently forward a copy of a completed cost-benefit analysis to DCS.

DCS considers that the review of cost-benefit analyses is not its responsibility. Information provided by DCS publicly on requirements for campaigns over $1 million links readers to the NSW Government's Cost Benefit Analysis Framework and User Guide, and directs readers to NSW Treasury for further information.

Previous government advertising audits have found that agencies have not completed a cost-benefit analysis before a campaign has begun or have otherwise not satisfactorily completed documentation required under the Government's advertising regulatory framework:

  • Our Government advertising: 2018–19 and 2019–20 report found that the agencies subject to the audit, Service NSW and the NSW Rural Fire Service, had prepared cost-benefit analysis which considered an alternative to advertising but did not assess the costs and benefits of the alternative. The same report also highlighted two previous audits that had found the requirement to prepare a cost-benefit analysis had not been fully met. These campaigns were delivered.
  • Our Government advertising 2020–21 report found that the agency subject to the audit, Destination NSW, did not prepare a cost-benefit analysis and the Acting Chief Executive signed a compliance certificate before the campaigns were planned, designed, or conceived. The campaign was delivered.

Cost-benefit analysis is a tool that improves the ability of policy makers and government to be confident that advertising expenditure is consistent with NSW Government requirements. The absence of processes to ensure that agencies are compliant in undertaking this analysis limits the ability to offer assurance that campaigns are an efficient, effective, and economical use of funds.

2.2 Compliance with the Guidelines and relevant policy

TAFE NSW met the recommended minimum spend of their media budget for communications with CALD and Aboriginal audiences

The NSW Government policy requires that government advertising must be sensitive to cultural needs and issues, and reflect the cultural and linguistic diversity of NSW. It also requires that at least 7.5% of an advertising campaign media budget be spent on direct communications to multicultural and Aboriginal audiences. Spend may be on media or non-media communication activities (for example, events, participation at cultural festivals, direct mail, competitions, and websites).

TAFE NSW’s planned media budget for direct communications in 2021–22 was $12.6 million, of which $1 million was intended for media directly targeted to multicultural and Aboriginal audiences. This represented a budget for CALD and Aboriginal audiences of 7.9%.

In 2021–22, TAFE NSW's actual spend on advertising to these audiences was slightly more than planned (at $1.02 million) and its actual overall spend on media advertising was slightly less than planned (at $12.04 million). As a result, TAFE NSW spent 8.5% of its total spend on media advertising to multicultural and Aboriginal audiences.

The campaign complied with the NSW Government’s evaluation framework and brand guidelines

TAFE NSW prepared a high-level evaluation framework, which was consistent with expectations set by DCS and Section 5.4 of the Government Advertising Guidelines. TAFE NSW also has a ‘brand book’ that establishes rules and guidance for the use of TAFE NSW’s logo and the appropriate presentation of website and other advertising material. The brand book includes guidance that aims to ensure all communications are accessible to people who have disabilities.

TAFE NSW’s advertising did not always comply with requirements to present information accurately and objectively

The audit recognises and acknowledges that TAFE NSW operates in a competitive environment and needs to market its educational services to current and prospective students. Further, TAFE NSW's advertising material might not obviously look like advertising delivered by the government. However, to comply with the Government's advertising guidelines, TAFE NSW's advertising must be evidence based.

TAFE NSW’s website asserts that it is the ‘leading provider of vocational education and training in Australia’ and its strategic plan for 2022–25 also claims that TAFE NSW is ‘Australia’s leading training provider’. TAFE NSW states that it is Australia’s largest vocational education training provider and that this justifies its use of the adjective ‘leading’.

The audit considers that TAFE NSW requires more evidence than its market share to substantiate its claim to be ‘Australia’s leading training provider’, as ‘leading provider’ also implies the top-ranked or objectively ‘best’ training provider.

TAFE NSW provided evidence that it has a history of winning the NSW Department of Education's 'large training provider of the year' award, although it last won this award in 2018. Nationally, it last won the corresponding award in 2013.

3. Effectiveness, efficiency and economy of the campaign

This part of the report considers whether TAFE NSW's advertising program for 2021–22 was carried out in an effective, efficient, and economical manner.

3.1 Campaign effectiveness

TAFE NSW established relevant objectives and measures for the campaign

The NSW Government ‘Evaluation Framework for Advertising and Communications’ requires advertising campaigns to be measured against their:

  • outputs, or what is done that reaches and engages the target audience. Outputs are assessed using metrics of exposure and reception
  • outcomes, or what the target audience takes from the communication and what sustainable effect the communication has on the audience. Outcomes are measured by the consumer response to the campaign
  • impacts, or the results that are caused by the campaign. Impacts are measured by objective actions or events.

TAFE NSW complied with the NSW Government evaluation framework. It prepared a one-page summary of the campaign’s objectives and its links to organisational and government objectives, the target audience and key messages, and the planned inputs, activities, outcomes and impacts. TAFE NSW also prepared a ‘Campaign Objectives Table’ which sets out detailed information on baseline and targeted performance, measurement milestones and evaluation methods. Measures can be evidenced and are linked to the campaign objectives. Exhibit 4 summarises TAFE NSW’s measurement of its annual campaign.

Exhibit 4: Evaluation measures for TAFE NSW’s annual advertising campaign for 2020–21
Type of measure Number of measures Example measures
Output 32

Number of radio, television, internet advertisements.

Cost per click through from website advertising.

Outcome 10

Number of website visits and course enquiries.

Percentage of target audience agreeing with statements about the advertising's believability, emotional engagement, relevance etc.

Number of student enrolments.

Impact 4 Percentage of target audience considering TAFE enrolment.
Source: Audit Office analysis based on information provided by TAFE NSW.

TAFE NSW achieved slightly over half of its output and outcome targets, and met all impact targets

The campaign met half of the output targets, seven of ten outcome targets and all four of the impact targets. The campaign only narrowly failed to achieve some of the intended output and outcome targets.

While there is some natural variability in the achievement of advertising campaign targets, TAFE NSW’s performance was affected by changes in the education and advertising marketplaces during 2021–22. For example, lockdowns as part of the response to the COVID-19 pandemic led to the closure of cinemas and some newspapers which, in turn, impacted TAFE NSW’s intended levels of cinema and print advertising.

TAFE NSW also adjusted its investment in various advertising channels throughout the year in response to changes in:

  • business performance measures, such as student enrolments in specific localities
  • operational performance measures, such as the extent to which some digital advertising resulted in too many users rapidly leaving the website, referred to as a high ‘bounce factor’.

Some of these adjustments resulted in lower achievement of output measures.

Exhibit 5 summarises TAFE NSW’s achievements against planned outputs, outcomes, and impacts. 

Exhibit 5: TAFE NSW campaign results
Type of measure Number of measures Results Comments
Output 32 16 The campaign met one-half of the output measures. Contributing factors to non-achievement included changes to the planned activity led by TAFE NSW’s in-year assessment of what will drive enrolments, as well as other changes in response to commercial and operational pressures including the COVID-19 pandemic and natural events such as floods.
Outcome 10 7 The campaign met most outcome measures but failed to meet several measures including the targets for ‘how likely to visit website after seeing the advertisement’ (a result of 61% against a target of 64%). The campaign also achieved 15.4 million website visits which was 4.7% below the target of 16.2 million visits.
Impact 4 4 The campaign met all key impact measures including building awareness of TAFE and the percentage of the target audience considering TAFE for prospective enrolment.
Source: Audit Office analysis.

There is a risk that some of TAFE NSW's results are overstated because it evaluated performance by surveying a population that was significantly different to its target audience

TAFE NSW’s advertising campaign has a range of target audiences which depend on the portfolio of courses that are advertised. However, in general, the campaign targets four distinct audiences which TAFE NSW defines as:

  • pathfinders, or people aged 15 to 19
  • career starters, or people aged 20 to 24
  • career progressors, or people aged 25 to 44
  • career changers, or people aged 45 years and over.

TAFE NSW evaluates the effect of its advertising using a range of performance data, part of which requires a twice-yearly survey of consumer preferences and views. In 2022, TAFE NSW surveyed 987 consumers to track its brand performance and evaluate its advertising campaign. TAFE NSW chose these consumers to reflect the age profile of New South Wales residents aged between 16 and 60. As a result:

  • 20% of survey respondents were aged 16 to 24 despite TAFE NSW directing 45% of its planned spending to people in that age group
  • 33% of respondents were aged 45 to 60 despite TAFE NSW targeting only 10% of planned spend at this age group.

Exhibit 6 summarises TAFE NSW's planned advertising investment for 2021–22 against each of its target audiences and the contribution that these age groups made to TAFE NSW's evaluation of the effectiveness of its overall advertising. 

Exhibit 6: TAFE NSW’s planned investment and evaluation by audience segment
Audience segment Planned
spend
(%)
Proportion of
valuation score
(%)
Comments
Pathfinders
(16 to 19)
15 8 Views of pathfinders are under-represented in evaluation.
Career starters
(20 to 24)
30 12 Views of career starters are under-represented in evaluation.
Career progressors
(25 to 44)
45 47 Very similar investment and evaluation proportions.
Career changers
(45 to 60)
10 33 Views of career changers are over-represented in evaluation.
Source: Audit Office analysis.

TAFE NSW states that it has used the same measurement approach since 2018 and considers that the approach provides insight into TAFE NSW's brand performance more generally. It further states that the survey's proportion of people over 40 is appropriate as these people will frequently influence education decisions made by younger cohorts, particularly those aged 16 to 19.

During this audit, TAFE NSW re-calculated its survey results from June 2022 using an age profile that reflected its current student population (which is significantly younger than New South Wales general population and is closer to, but not the same as, TAFE NSW's planned spend by audience segment). TAFE NSW stated that the re-calculated survey found the proportion of people who would consider TAFE NSW for future study was reduced from 69% to 65%. 

3.2 Campaign efficiency and economy

The absence of a cost-benefit analysis for this campaign means that the investment decision was made without a clear view of potential value for money

TAFE NSW’s approved advertising campaign budget was $15.25 million, which significantly exceeded the legislative threshold of $1 million that requires a cost-benefit analysis to be carried out before the start of the campaign. While TAFE NSW prepared a rationale to support its advertising investment, it did not prepare the required cost-benefit analysis for its 2020–21 campaign.

It is appropriate that government is fully informed of the potential value for money of an advertising campaign before it decides to invest in the campaign. The Act requires that this should be done by the completion of a cost-benefit analysis. Robust analysis of the potential costs and benefits of an advertising campaign may also improve the evaluation’s design and process.

TAFE NSW’s annual advertising campaign was adjusted throughout the year in response to business data, which is likely to have led to efficient expenditure

Consistent with other parts of government, TAFE NSW prepares an annual advertising plan that sets out advertising and search engine optimisation (which is the set of techniques used to improve the likelihood of internet users finding the advertiser’s website or product when they search using any of the standard internet search tools). The plan is reviewed by DCS to ensure the planned advertising has evidence of its likely effectiveness, leverages government communication channels, and makes use of data and customer and behavioural insights. Once approved by ERC, TAFE NSW works with its media partner to execute the advertising plan by placing media throughout the year. However, TAFE NSW also adjusts its investment in various advertising channels throughout the year in response to changes in:

  • business performance measures. For example, lower-than-expected student enrolments in specific localities may lead to TAFE NSW running a localised advertising campaigns during the year to boost student enrolments
  • operational performance measures. For example, data from a pilot showed that some digital advertising that TAFE NSW had planned to run throughout 2021–22 was resulting in too many users clicking through to a website but then rapidly leaving it (this is known as a high ‘bounce factor’). TAFE NSW stopped the planned digital advertising and redirected the planned expenditure to other advertising channels.

The combination of a peer-reviewed overall annual advertising plan and appropriate adjustments to events and data throughout the year means that TAFE NSW’s spend on advertising is likely to be efficient.

TAFE NSW’s campaign is likely to have been economical as it used in-house capability to develop the creative content and then relied upon whole-of-government arrangements to ensure the prices it paid for advertising were market rates

TAFE NSW spent just over 79% of its total advertising spend in 2021–22 to buy advertising media, such as social media advertisements, out of home advertising, and print advertising. The remainder of its total advertising spend was on research and evaluation, planning, and service fees.

TAFE NSW used its media partner to arrange and place this media, using prices which had been negotiated by another communications agency on behalf of all NSW Government with media providers such as newspapers, television, radio and internet companies. TAFE NSW developed the creative content for the campaign, with still photography and video shot on campus using staff and students in normal contexts.

In 2017, the Department of Premier and Cabinet’s communications directorate (now located within DCS) chose TAFE NSW's media partner to be one of five providers of communication services that could be used by government agencies to buy media. Under that arrangement, one of these providers (which was not TAFE NSW's partner) was also responsible for annually negotiating market rates on behalf of government as well as other responsibilities such as whole-of-government reporting. The arrangement defined these prices as ceiling rates and provided the opportunity for the five providers of communication services to negotiate better rates which may become available due to competitive pressures in the media market.

The advertising rates under the whole-of-government contract were reviewed annually throughout the life of the contract and were independently market-tested in 2018–19 by DCS. TAFE NSW also states that some of its advertising spend was at prices significantly lower than the whole-ofgovernment contract. As a consequence, the prices paid by TAFE NSW for advertising in 2021–22 were likely to have been market rates.

Appendices

Appendix one – Responses from agencies

Appendix two – About the campaign

Appendix three – About the audit

Appendix four – Performance auditing

 

Copyright notice

© Copyright reserved by the Audit Office of New South Wales. All rights reserved. No part of this publication may be reproduced without prior consent of the Audit Office of New South Wales. The Audit Office does not accept responsibility for loss or damage suffered by any person acting on or refraining from action as a result of any of this material.

 

Parliamentary reference - Report number #377 - released 28 February 2023