What the report is about
The Government Advertising Act 2011 requires the Auditor General to conduct a performance audit on government advertising activities each financial year.
This audit looked at whether three campaigns run by Destination NSW (DNSW) during 2020–21 were carried out in an effective, economical and efficient manner:
- Love Sydney (comprising two sub campaigns being ‘Sydney - Love It Like You Mean It’ and ‘Get Your Sydney On’)
- Love NSW
- Road Trips.
What we found
DNSW complied with section 6 of the Government Advertising Act 2011 (the Act), prohibiting political content.
The Act requires the head of an agency to sign a compliance certificate that certifies that the campaign complies with the Act and is an efficient and cost effective means of achieving its public purpose.
When the Acting Chief Executive of DNSW signed DNSW’s compliance certificate, evidence to support this certification was not available.
The Act requires a peer review and cost benefit analysis for campaigns over $1.0 million. DNSW did not complete the peer review or cost-benefit analysis for the audited advertising campaigns before they had concluded.
The Department of Customer Service (DCS), which manages the peer review process, did not escalate the issue of the outstanding peer review documentation to senior DNSW staff.
DNSW did not set targets for all measures established for the campaigns. This limits the ability to assess their effectiveness.
The impact of the COVID-19 pandemic likely contributed to the campaigns not meeting a substantial proportion of established outcome and impact targets.
None of the audited campaigns met the minimum requirement of 7.5 per cent for the allocation of the media budget for communications with Culturally and Linguistically Diverse and Aboriginal audiences.
What we recommended
- implement processes for planning and delivering advertising campaigns delivered in urgent circumstances to bring them in line with NSW Government practice
- ensure that it establishes measurements and targets for outcomes and impacts of its advertising campaigns consistent with NSW Government evaluation frameworks and guidance.
The Department of Customer Service should:
- establish a policy and procedure for ensuring that campaign documentation is completed in a timely manner in the case of urgent campaigns, including establishing expectations around timeframes for the completion of peer review
- establish a procedure for escalating issues of outstanding documentation to ensure that the peer review is completed in line with reasonable expectations and timeframes.
- $9.6m is the total money spent on the three audited campaigns
- $91.2m is the total amount of money spent by the NSW Government on advertising in 2020–21.
The Government Advertising Act 2011 (the Act) requires the Auditor-General to conduct a performance audit on the activities of one or more government agencies in relation to government advertising campaigns in each financial year. The performance audit assesses whether a government agency or agencies have carried out activities in relation to government advertising in an effective, economical and efficient manner and in compliance with the Act, the regulations, other laws and the Government Advertising Guidelines (the Guidelines). This audit examined three campaigns run by Destination NSW during the 2020–21 financial year:
- Love Sydney (comprising two sub-campaigns being ‘Sydney - Love It Like You Mean It’ and ‘Get Your Sydney On’), focussing on increasing visitor activity in Sydney
- Love NSW, focussing on increasing visitor activity in regional New South Wales
- Road Trips, focussing on encouraging visitor activity on iconic road trips in regional New South Wales.
Section 6 of the Act prohibits political advertising. Under this section, material that is part of a government advertising campaign must not contain the name, voice or image of a minister, member of Parliament or a candidate nominated for election to Parliament or the name, logo or any slogan of a political party. Further, a campaign must not be designed to influence (directly or indirectly) support for a political party.
The Act and associated regulations and the Guidelines also establish an accountability and compliance framework around the investment in advertising by NSW Government agencies.
The government's operating circumstances at the commencement of the 2020–21 financial year were highly challenging, with the 2019–20 bushfires being followed by the COVID-19 pandemic. This created new demands across a range of government services, and without any clear view on the severity of the pandemic and when it would end. This was the case for Destination NSW, which had to plan for its advertising activities in the context of an uncertain future for national border closures (impacting international in-bound travel) and lockdowns across Australia, including in New South Wales (impacting domestic travel). Further, the sudden nature of outbreaks and lockdowns meant that Destination NSW often was required to change the targeting of its campaigns and, in some situations, had to cease particular advertising activities until specific lockdowns had ended.
The three Destination NSW campaigns subject to this audit were consistent with the allowed purposes of government advertising and did not include political advertising.
Destination NSW did not comply with the requirement to complete a peer review of campaigns, nor did it complete a cost-benefit analysis before or during the conduct of each of the audited campaigns. These requirements of the Act are designed to provide reasonable assurance that the advertising campaigns represented efficient, effective and economical uses of government funds.
Two of the three campaigns achieved some of their objectives relating to influencing consumers. The effects of the COVID-19 pandemic likely contributed to all of the campaigns not meeting a substantial proportion of established outcome and impact targets, with the impact of COVID-19 varying across campaigns and performance measures. It is particularly difficult to determine the impact of COVID-19 where measures or targets have not been set, as was the case with some of the measures for these campaigns. The impact of the COVID-19 pandemic also meant Destination NSW needed to make media placement changes when lockdown resulted in pauses or re-directions of media activities. This led to some unforeseen expenditure, but was an unavoidable consequence of needing to make changes at short notice.
Destination NSW was only able to present evidence that two of the campaigns ('Sydney - Love It Like You Mean It' and 'Love NSW') represented a positive benefit-cost ratio.
The Act requires the head of an agency to sign a compliance certificate stating that, among other things, the campaign complies with the Act, the regulations and the Guidelines, and that the campaign is an efficient and cost-effective means of achieving the public purpose. The Acting Chief Executive of Destination NSW signed the required compliance certificate associated with all of its 2020–21 advertising campaigns in February 2020, before they had been designed and planned, and before the associated expenditure had been approved.
Destination NSW did not complete required cost-benefit analyses before the campaigns commenced or while the campaigns were airing and did not establish complete suites of measures and targets for impact and outcomes of the advertising campaigns to inform the campaign.
Destination NSW did not ensure that the required peer review process was completed in a timely manner. The Department of Customer Service (DCS) supported Destination NSW's decision to commence the campaigns while the peer review was completed simultaneously. The Act allows this for urgent campaigns, and Destination NSW and DCS agreed that the need for this campaign to support driving economic activity in New South Wales after months of reduced activity brought on initially by the 2019–20 bushfires and then by the pandemic warranted this approach. As the campaigns progressed, DCS provided reminders to complete the peer review process, but this was not done. DCS did not escalate the issue of the incomplete peer review during this time. In September 2021 it advised Destination NSW officially that it would not consider further submissions for peer review with regard to the completed campaigns.
Destination NSW could not demonstrate how its campaign designs or media placements effectively supported the cultural needs and issues of culturally and linguistically diverse populations, consistent with the requirements of the 'Culturally and Linguistically Diverse (CALD) and Aboriginal Advertising Policy'.
Destination NSW did not establish comprehensive suites of measures and targets to allow for robust assessments of whether the campaigns achieved the intended outcomes from the campaigns. This limited the effectiveness of these measures as an accountability tool as intended by the NSW Government evaluation framework.
By 30 June 2022, Destination NSW should:
1. implement processes for planning and delivering advertising campaigns delivered in urgent circumstances to bring them in line with NSW Government practice by ensuring:
- peer reviews are completed prior to the end of the campaign
- cost-benefit analyses are completed prior to the end of the campaign
- compliance with the Culturally and Linguistically Diverse (CALD) and Aboriginal Advertising Policy
- the Chief Executive's compliance certification required by the Act is based on substantially completing the compliance obligations in advance of the campaign's commencement.
2. ensure that it establishes measurements and targets for outcomes and impacts of its advertising campaigns consistent with NSW Government evaluation frameworks and guidance.
By 30 June 2022, the Department of Customer Service should:
3. establish a policy and procedure for ensuring that campaign documentation is completed in a timely manner in the case of urgent campaigns, including establishing expectations around timeframes for the completion of peer review
4. establish a procedure for escalating issues of outstanding documentation to ensure that the peer review is completed in line with reasonable expectations and timeframes.
What is government advertising?
Governments use advertising to communicate information about a government program, policy or initiative to members of the public. Government advertising is funded by or on behalf of a government agency and is disseminated under a commercial advertising distribution agreement through a variety of media, such as radio, television, the Internet, newspapers, billboards or cinemas.
There are three broad categories of government advertising:
- recruitment advertising - advertising which promotes specific job vacancies and employment opportunities within a government agency
- public notices - advertising which communicates a clear, simple message or announcement and is generally one-off or short-term in nature
- public awareness advertising - co-ordinated communications to raise awareness of key issues, such as government initiatives, or social marketing to encourage behaviour change.
How much is spent on government advertising in New South Wales?
The NSW Government spent $91.2 million on advertising in 2020–21. The overall expenditure for the five previous financial years may be seen in Exhibit 1. Note that in the 'Government advertising 2018–19 and 2019–20' performance audit the figure stated for 2018–19 expenditure was $83.8 million. The correct figure is $91.7 million, which is reflected in the below graph.
How is government advertising regulated?
A regulatory framework which includes both policy and legislation governs NSW Government advertising as set out in Premier's Memorandum M2012-12 Government Advertising Reform. Exhibit 2 contains an overview of this regulatory framework.
|Government Advertising Act 2011 (Act)
|Sets out the legal requirements for government advertising.
|Government Advertising Regulation 2018 (Regulation)
|Sets out exemptions to the Act.
|NSW Government Advertising Guidelines 2012 (Guidelines)
|Sets out requirements in relation to the style and content, dissemination and cost of government advertising campaigns, as well as the requirements of cost-benefit analyses and peer reviews. The Guidelines were issued by the then Premier of NSW.
Prohibition of political advertising
Section 6 of the Act prohibits political advertising as part of a government advertising campaign. Government advertising campaigns must not:
- be designed to influence (directly or indirectly) support for a political party
- contain the name, voice or image of a minister, a member of Parliament or a candidate nominated for election to Parliament
- contain the name, logo, slogan or any other reference to a political party.
In addition, the Guidelines require government advertising campaigns to be politically neutral and clearly distinguishable from party political messages.
Requirements prior to the commencement of a campaign
The Act states that a government advertising campaign must not commence unless the head of the agency has signed a compliance certificate for the campaign. This compliance certificate states that the head of the agency believes the government advertising campaign:
- complies with the Act, Regulation and Guidelines
- contains accurate information
- is necessary to achieve a public purpose and is supported by analysis and research
- is an efficient and cost-effective means of achieving its public purpose.
The Act defines further requirements for campaigns which are likely to exceed a total cost of $250,000 and $1.0 million. These are summarised in Exhibit 3.
Government advertising campaigns likely to cost over $250,000 are subject to peer review before the campaign commences.
The Department of Customer Service (DCS) manages the peer review process. The peer review process involves DCS employees assessing the proposed advertising campaign according to a set of criteria outlined in the Guidelines. Peer review is a two-stage process:
- Budget approval - government agencies submit an advertising budget proposal to the DCS Campaign Effectiveness Team. The Campaign Effectiveness Team reviews the budget proposal and makes recommendations to the Delivery and Performance Committee of Cabinet (DaPCo). DaPCo is responsible for approving campaign budgets.
- Campaign review - after DaPCo approves a campaign budget, the DCS Campaign Effectiveness Team conducts a review of the proposed campaign. The Campaign Effectiveness Team assesses the need for the proposed advertising campaign, the creative and media strategy (including objectives and target audiences) and how the agency will manage the campaign. Specifically, the campaign review considers:
- alignment with government priorities and commitments and the life of New South Wales customers
- effectiveness of approach to regional customers, Aboriginal customers and Culturally and Linguistically Diverse (CALD) customers
- creative strategy and its alignment to campaign objectives
- media strategy and its alignment to campaign objectives, including effectiveness of consideration of digital channels
- how the advertising integrates with other communication approaches
- clarity and appropriateness of output, outcome and impact objectives, measures and targets.
Peer reviews are an important step in a campaign as they provide independent assurance for the agency regarding the campaigns' compliance with some legislative and policy obligations, that government advertising campaigns are necessary, and that they are established with suitable objectives, measures and targets.
The Act requires an agency proposing to run a government advertising campaign that is likely to cost over $1.0 million to complete a cost-benefit analysis before the campaign commences. A cost-benefit analysis aims to evaluate the net economic cost or benefit of the campaign and indicate to decision-makers how the campaign will affect the wellbeing of New South Wales residents. It also plays an important role as a baseline for post-campaign evaluation.
|Budget approval by DaPCo
|$250,000 to $1.0 million
1.2 About Destination NSW
Destination NSW is a Public Sector Executive Agency responsible for devising and implementing strategies to grow the visitor economy. The principal objective of Destination NSW is to achieve economic and social benefits for the people of New South Wales through the development of tourism and the securing of major events.
Destination NSW has established as its mission to triple overnight visitor expenditure by 2030 and maximise the benefits of the visitor economy for New South Wales. To support this, it published a Visitor Economy Strategy 2030 in January 2021, which outlines a range of activities to achieve this mission. One of those activities is to 'promote Sydney and New South Wales through striking new marketing campaigns, world class events, and a renewed focus on the 24-hour economy'.
A key focus of the Visitor Economy Strategy 2030 is to 'ensure New South Wales recovers quickly from the impacts of COVID-19, bushfires, drought and floods and elevates its status as the premier visitor economy in Asia-Pacific'.
1.3 About the audited campaigns
The 2020–21 government advertising campaigns examined in this audit are listed in Exhibit 4.
|Love Sydney (comprising two sub-campaigns being ‘Sydney - Love It Like You Mean It’ and ‘Get Your Sydney On’)
The Love Sydney campaign was originally envisaged as part of a broad set of campaigns focussing on events and features of Sydney, which was intended to create and convert tourism demand for Sydney. With the ongoing pandemic and lockdowns in various parts of the country, its focus changed to create more immediate leisure activity from residents in Sydney to respond to the significant downturn in economic activity brought about by the pandemic.
The ‘Sydney - Love It Like You Mean It’ campaign was focussed on residents in the greater Sydney region, encouraging immediate action to plan and book activities in their own city to support attractions, retail and hospitality venues, museums and galleries. The campaign included radio, television, print, out-of-home (billboard) and digital advertising. Advertising activities as part of this campaign occurred from October 2020 to June 2021.
The ‘Get Your Sydney On’ campaign was focussed on residents of South East Queensland, Victoria, Australian Capital Territory and regional New South Wales. Its objectives were also to encourage immediate action to plan and book travel to Sydney to support attractions, retail and hospitality venues, museums and galleries at a time when there was an expectation of borders opening up and interstate travel occurring. The campaign included radio, broadcast video on demand and social media. Advertising activities as part of this campaign occurred from April to June 2021.
Both campaigns were impacted by COVID-19 outbreaks in Sydney, which caused pauses in advertising, as well as State border closures, particularly on the borders with Victoria and Queensland.
The Love NSW campaign was the second phase of a campaign which commenced in 2019–20. Phase two was launched in New South Wales in August 2020 and then expanded to other states as borders were reopening in the second half of 2020. The campaign ran as a series of components encouraging travel and activities for different purposes within regional New South Wales (including around school holidays or around specific border openings). The campaign also focussed on specific regions of New South Wales that had been particularly impacted by a downturn in activity.
Advertising activities include digital advertising, print media, sponsored content on certain websites and television in certain regions. The campaign operated from August to December 2020, and late January to April 2021. Breaks and media re-directions were required due to COVID-19 outbreaks.
The Road Trips campaign also focussed on regional New South Wales and concentrated on travel by vehicle within the State. The campaign involved extensive production activity, including following Australian celebrities on iconic journeys in New South Wales. The Road Trips campaign was heavily digitally focussed, and also included partnerships with media companies. The campaign ran from December 2020 to June 2021.
1.4 About this audit
The Act requires the Auditor-General to conduct a performance audit on the activities of one or more government agencies in relation to government advertising campaigns in each financial year.
In conducting the audit, the Auditor-General must determine whether a government agency or agencies have carried out activities in relation to government advertising in an effective, economical and efficient manner and in compliance with the Act, the Guidelines and other laws.
Appendix two contains details about each campaign and Appendix three contains further details about this audit.
The audit team did not review the broader use of Destination NSW's social media outside paid social media content relevant to these campaigns. Unpaid social media content is not considered government advertising for the purposes of the Act.
2. Compliance with the Government Advertising Act 2011, the regulations, guidelines and other laws
All three advertising campaigns complied with the political advertising prohibitions in the Act and were for an allowed purpose.
The Acting Chief Executive of Destination NSW signed the required compliance certificate associated with all of its 2020–21 advertising campaigns in February 2020, before the campaigns had been designed and planned, and before the associated expenditure had been approved. This means that the assertions in the certification could not be supported. It is therefore not a reliable certification of compliance with the Act. A more reliable approach to completion of the compliance certificate, and an approach that is more typical across other NSW Government advertising campaigns, is to complete the certification after all planning and designs work is done, after the peer review is complete, and immediately prior to the launch of the campaign.
Destination NSW did not complete the peer review of campaigns, nor a cost-benefit analysis before or during the conduct of the audited campaigns. This is inconsistent with key aspects of accountability within the NSW Government's framework for advertising. As the campaigns progressed, DCS provided reminders to complete the peer review process, but this was not done by Destination NSW prior to the end of the campaigns. DCS did not escalate the issue of the incomplete peer review during this time. In September 2021 DCS advised Destination NSW officially that it would not consider further submissions with regard to the completed campaigns.
Destination NSW could not demonstrate how its campaign designs or media placements effectively supported the cultural needs and issues of culturally and linguistically diverse populations, consistent with the requirements of the 'CALD and Aboriginal Advertising Policy'.
Campaign materials we reviewed did not contain political content
The audit team reviewed campaign materials developed as part of each of the paid advertising campaigns including radio transcripts, digital videos and display. See Appendix two for examples of campaign materials for this campaign.
Section 6 of the Act prohibits political advertising as part of a government advertising campaign. A government advertising campaign must not:
- be designed to influence (directly or indirectly) support for a political party
- contain the name, voice or image of a minister, a member of parliament or a candidate nominated for election to parliament
- contain the name, logo, slogan or any other reference to a political party.
The audit found no breaches of section 6 of the Act in the campaign material reviewed.
All reviewed campaigns were for purposes permitted by section 1.2 of the Guidelines
Section 4 of the Act states that government advertising campaigns are 'the dissemination to members of the public of information about a government program, policy or initiative, or about any public health or safety or other matter'. To support this, section 1.2 of the NSW Government Advertising Guidelines states that government advertising campaigns may only be used to achieve certain objectives. One of these objectives is to encourage changed behaviours or attitudes that will lead to improved public health and safety or quality of life.
The audit team considers that each of the reviewed advertising campaigns was consistent with this objective. This reflects the intent of each of the campaigns to increase economic activity driven by tourism activity in New South Wales, that contributes to improved quality of life for New South Wales residents.
The Acting Chief Executive signed Destination NSW's compliance certificate without supporting evidence
The Acting Chief Executive of Destination NSW signed a single compliance certificate for all Destination NSW campaigns for 2020–21 (including the three campaigns that are considered by this audit) on 28 February 2020. Evidence was not available at this date to support the statements included in the compliance certificate for the campaigns that were considered by this audit.
The compliance certificate is required by section 8 of the Act and states that the head of the agency confirms that a proposed government advertising campaign:
- complies with the Act, the regulations and the Guidelines, and
- contains accurate information, and
- is necessary to achieve a public purpose and is supported by analysis and research, and
- is an efficient and cost-effective means of achieving that public purpose.
At the time of signing the certificate in February 2020, Destination NSW had not conceived, designed or planned any of the campaigns that are considered by this audit, nor had it developed the relevant supporting information that would enable the agency to support these statements. As noted above, peer review had not commenced prior to this date. Further, Destination NSW had not completed a cost-benefit analysis or equivalent analysis.
Without any form of cost-benefit analysis or other evaluation for any of the campaigns prior to the date of signing of the compliance certificate, the Acting Chief Executive had no evidence that could support the certification that the campaigns were 'an efficient and effective means of achieving the public purpose'. The absence of peer review or a cost-benefit analysis also means that the Acting Chief Executive could not certify that the campaigns complied with the Act, the regulations or the Guidelines, nor that the campaign was supported by analysis and research.
Destination NSW did not complete peer reviews for the advertising campaigns before they ended, limiting assurance over campaign effectiveness, efficiency and economy
As all the campaigns subject to this audit were valued at over $250,000, each campaign was required to undergo peer review. The peer review is an independent review of the need for the proposed advertising campaign, the creative and media strategy (including objectives and target audiences) and how the agency will manage the campaign. Ordinarily, a peer review would be completed prior to a campaign commencing, however section 7(4) of the Act permits agencies to carry out a peer review after the advertising campaign commences 'if the head of the government agency concerned is satisfied that the campaign relates to an urgent public health or safety matter or is required in other urgent circumstances'.
DCS supported Destination NSW's assessment that these were urgent campaigns and that it would accept consideration of peer review components in parallel with the roll-out of the advertising campaigns, given the urgency of the need to generate economic activity, initially after the 2019–20 bushfires and then after the challenging circumstances brought on by the COVID-19 pandemic. This is in line with section 7(4) of the Act.
Destination NSW presented and obtained clearance on creative materials and media planning on a timely basis for two of the three campaigns (but not for the Road Trips campaign), which would ordinarily form part of peer review. However, for all campaigns, the peer reviews were not completed or signed off by DCS prior to the completion of advertising campaigns. In particular, Destination NSW did not submit material related to the accountability for campaign effectiveness, including the campaign objectives and measures before the end of the campaigns.
The absence of peer review of much of the material prior to completion of the campaigns reduces the ability of the agency and government to be confident that the advertising expenditure was consistent with NSW Government requirements, or represented efficient, effective and economical use of funds.
Destination NSW noted that section 7(4) of the Act allows the peer review to be completed after the commencement of a campaign in urgent circumstances but places no requirement on it to be completed before the end of the campaign. The audit has determined that for the peer review to meet its intended purpose, being to inform the design and delivery of the advertising campaign, it needs to be completed prior to the end of the campaign, even in urgent circumstances. DCS has supported this intent of the framework.
By the end of September 2021, DCS advised Destination NSW that it would not consider any further material for peer review related to the 2020–21 advertising campaigns. At this time, DCS closed the peer review for the Love NSW and Road Trips campaigns and assessed them as incomplete. DCS assessed the Love Sydney peer review as complete, despite noting that the campaign evaluation was not complete and with no details or confirmation of meeting culturally and linguistically diverse (CALD) advertising requirements, including for Aboriginal communities.
DCS did not escalate the issue of outstanding peer review materials
DCS worked at officer level to remind Destination NSW that peer review material was outstanding during the year. While this is appropriate as an initial point of escalation, at no time was the issue of non-compliance escalated to higher levels of management. DCS also never sent formal correspondence requesting the materials needed to ensure the completion of peer review.
DCS does not have a process for ensuring the timely completion of peer review in situations where urgency exemptions are used. There is an opportunity to formalise this process to ensure that there are appropriate escalation points and to ensure that compliance obligations are fulfilled in future.
Destination NSW did not meet the minimum requirement for allocation of the media budget for communications with CALD and Aboriginal audiences
The NSW Government 'CALD and Aboriginal Advertising Policy' stipulates that at least 7.5 per cent of an advertising campaign media budget is to be spent on direct communications to multicultural and Aboriginal audiences. Spend may be on media or non-media communication activities (e.g. events, participation at cultural festivals, direct mail, competitions and websites).
Destination NSW spent only 1.6 per cent of its media spend on culturally and linguistically diverse specific media placement on the 'Sydney - Love It Like You Mean It' campaign and none of its media placement for the other audited campaigns. This level of expenditure is substantially below the requirement.
Destination NSW could not demonstrate how its campaign designs or media placements effectively supported the cultural needs and issues of culturally and linguistically diverse populations. In connection with the 'Sydney - Love It Like You Mean It' campaign, it was noted that timeframes and production issues limited the ability to incorporate culturally diverse individuals in imagery.
Destination NSW advised that it believes the application of a 7.5 per cent threshold for specific audiences is not an effective way to reach these audiences. Destination NSW advised that its advertising was targeted at audiences with a propensity to travel, which did not necessarily include culturally diverse audiences, and its media channel research influenced its decision not to target specific CALD-focussed media channels.
None of the above factors negate Destination NSW's responsibility to ensure that the 'CALD and Aboriginal Advertising Policy' requirements are met.
In addition, Destination NSW also noted a number of non-media activities that supported culturally and linguistically diverse audiences, including translations on the sydney.com website, capturing of culturally and linguistically diverse audiences in production shooting and the production of a range of other collateral for culturally and linguistically diverse audiences. Despite these non-media activities, which Destination NSW did not quantify, the requirement for minimum expenditure in the reviewed campaigns for CALD audiences was not met by Destination NSW.
Destination NSW advised that it believes that the 7.5 per cent requirement does not apply to advertising outside of New South Wales, which the 'Get Your Sydney On', Love NSW and Road Trips campaigns targeted in whole or in part. The 'CALD and Aboriginal Advertising Policy' does not specifically limit its application to advertising for New South Wales residents.
3. Effectiveness, economy and efficiency of delivery of the campaigns
Destination NSW did not establish comprehensive suites of measures and targets to allow for robust assessments of whether the campaigns achieved the intended outcomes from the campaigns. This limited the effectiveness of these measures as an accountability tool as intended by the NSW Government Evaluation Framework.
None of the campaigns met the majority of the targets which had been established. This means that the campaigns did not have the market impact that was committed at the time of making the investment. Despite this, the Love NSW campaign did have a positive return on investment. The 'Get Your Sydney On' campaign was not required to undergo a cost-benefit analysis as it fell below the threshold, and the Road Trips campaign had not been assessed for return on investment at the time of the audit. This indicates a measure of cost-efficiency in the delivery of one of the campaigns, and a positive impact on the New South Wales economy. For the 'Sydney - Love It Like You Mean It' campaign, both the benefit-to-cost ratio and the return on investment were considerably below reasonable benchmarks, indicating a poor cost-efficiency outcome from the investment.
In all procurement of research, production and media services, Destination NSW complied with relevant procurement requirements, providing support to achieving value for money in relevant expenditure.
3.1 Campaign effectiveness
For any advertising campaign, there can be a range of measures of effectiveness. In this audit, campaign effectiveness has been assessed on the basis of achievement of actual results against measures and targets established early in the campaign. If the majority of targets for a campaign were achieved, then the campaign is assessed as achieving its objectives.
Destination NSW established relevant objectives and measures for all campaigns subject to this audit, however there were no outcome measures for one campaign
The NSW Government 'Evaluation Framework for Advertising and Communications' notes that there should be measures for advertising campaign outputs (a measure of exposure and reception to the campaign), outcomes (a measure of consumer response to the campaign) and impacts (a measure of results from the campaign).
For each of the campaigns, Destination NSW completed two documents required by DCS relevant to campaign objectives. The documents are:
- a 'Campaign Objectives Table' which seeks to outline the objectives for the advertising campaign
- an 'Evaluation Framework' template, which draws on the NSW Department of Premier and Cabinet’s 'Guidelines for Implementing the NSW Government Evaluation Framework for Advertising and Communications'. The Evaluation Framework seeks to present on one page the organisational and campaign objectives, the target audience and key messages for the campaign, and the key inputs, activities, outputs, outcomes and impacts that will form the basis for evaluation.
These Destination NSW documents were not developed when peer review commenced, and it is not clear when they were developed or what influence they had over the planning and delivery of the advertising campaigns. Over the course of each of the campaigns, the documented objectives (and in some cases the measures and targets) were refined after planning was completed and campaign deployment commenced.
The final campaign objectives for each of the campaigns are summarised in Exhibit 5 and a full list of objectives may be found in Appendix two.
|Sydney - Love It Like You Mean It
|Drive immediate action from Sydneysiders to plan and book activities in their own city to support businesses.
|Get Your Sydney On
|Drive immediate action to plan and book a break in Sydney.
|Assist the New South Wales visitor economy recovery via increasing awareness and appeal of regional New South Wales and increasing consideration and intention to take a regional New South Wales holiday.
|Increase market share of domestic road trips, increase average spend per visit per night, and encourage dispersal of visitation and spend into rural towns and businesses.
For all campaigns, there is a suite of quantifiable measures, which can be linked to the campaign objectives, and which can be evidenced. There is a particular emphasis on output measures, but three of the campaigns also had relevant impact and outcomes measures. The exception is 'Sydney - Love It Like You Mean It', which had no outcome measures.
Exhibit 6 sets out the number of measures established for each campaign in the Campaign Objectives Table. The key impact and outcome measures are outlined below the table.
|Sydney - Love It Like You Mean It
|Get Your Sydney On
High-level outcome objectives were described for the 'Sydney - Love It Like You Mean It' campaign in the evaluation framework, being to 'drive cut-through awareness and immediate action to encourage Sydneysiders to support Sydney businesses' and to 'increase gross economic benefit to Sydney, drive actual behaviour change and impact on local economy'. However, these were not translated to measurable items to drive accountability for those outcomes, which is required by the regulatory framework.
The impact measures for both of the Love Sydney campaigns sought to record the behaviour change arising from the campaign, including visiting a website about Sydney, considering Sydney for a short break, deciding to visit Sydney for a short break, and/or actually expanding a planned holiday or booking tickets to an event or attraction in Sydney. Both campaigns also established a measure of positive return on investment from the campaigns as part of these impact measures.
For the Love NSW campaign, the outcome measures included access to the visitnsw.com website and a conversion rate from these website visits, as well as measures of the increase in appeal and intention to visit NSW. The impact measures included measures of overnight visitor nights, visitor spend per night and average length of stay.
The outcome and impact measures for the Road Trips campaign were broadly similar to Love NSW, but also included measures of actual bookings arising from the campaign in the relevant regional areas.
Destination NSW did not set targets for all measures established for the campaigns, limiting accountability for the campaigns and the ability to assess their effectiveness
The 'Guidelines for Implementing the NSW Government Evaluation Framework for Advertising and Communications' includes a range of guidance for evaluation of advertising campaigns. Key to these is the use of the Evaluation Framework template described above. This matrix was completed for each of the in-scope campaigns.
Destination NSW established targets for all relevant measures for the 'Get Your Sydney On' and Love NSW campaigns. However, there were gaps in target setting for the impact measures for the 'Sydney - Love It Like You Mean It' campaign and the outcome and impact measures for the Road Trips campaign. The absence of quantifiable targets limits the accountability that comes with performance measurement and limits the ability for Destination NSW to assess the effectiveness of the campaigns.
Destination NSW advised that these campaigns had no precedent due to the impacts of COVID-19, which made target setting difficult. While this is true, the 'Guidelines for Implementing the NSW Government Evaluation Framework for Advertising and Communications' provide advice for target setting, including in contexts where there is an absence of precedence.
The 'Sydney - Love It Like You Mean It' campaign delivered some intended benefits, but not its core objective to drive immediate action to plan and book activities in Sydney
The 'Sydney - Love It Like You Mean It' campaign met the majority of its output measures. Although there were no measurable outcome targets, the campaign evaluation (which was based on a survey of a sample of consumers) indicates that its messages were broadly persuasive. In this context, persuasive means viewers believing a campaign's content and expressing interest in visiting the advertised location.
While the campaign did have some positive influence on consumers to undertake leisure activities in Sydney, it did not meet any of the impact targets that it established for the campaign, as can be seen in Exhibit 7 below.
|The advertising campaign achieved results in excess of the majority of targets against the range of reach and access measures established for the campaign. Four targets were partially met, and one was not met.
|In the absence of measures, it is impossible to assess achievement. However, the campaign evaluation noted: 'The campaign performed above the average of all previously tracked DNSW campaigns for involvement, salience and persuasion.' 71 per cent of evaluation respondents claimed to be persuaded by the messaging of the campaign to consider booking activities in Sydney.
|Did not achieve objective
|The evaluation results for 'taking action as a result of the campaign' was lower than the targets set by Destination NSW (for all measures where targets were set). The evaluation also demonstrated that, while the campaign was influential for residents under 45 years of age, a significant population (those over 45 year of age) were largely not impacted by the campaign.
It is possible that the actual performance being lower than some targets was a consequence of COVID-19 outbreaks and their influence on Sydney residents from undertaking increased leisure activities in Sydney.
The 'Get Your Sydney On' campaign did not meet its targets across outputs, outcomes or impact
The lockdowns arising from the COVID-19 pandemic had a particularly pronounced impact on the 'Get Your Sydney On' campaign, as it could only be in market for a fraction of the planned timeframe. This likely contributed to the campaign not meeting any of its outcome targets and only meeting one of its four impact targets.
The evaluation of the campaign commissioned by Destination NSW found that the campaign created some impact on people that recognised the campaign, with 24 per cent considering Sydney for a potential holiday or short break, 18 per cent making a decision to go to Sydney for a holiday or a short break, and nine per cent making a new booking in Sydney. However, each of these was lower than the targets established by Destination NSW for these measures.
|Did not achieve objective
|The campaign failed to meet the majority of reach and impressions targets.
|Did not achieve objective
|The campaign did not meet any of its outcome targets.
|Did not achieve objective
|The campaign did not meet any of its impact targets.
Consistent with the observation for the 'Sydney - Love It Like You Mean It' campaign above, it is impossible to conclude on the primary contributors to being unable to meet the majority of its campaign targets.
The Love NSW campaign did not meet its targets across most outputs and outcomes, and Destination NSW has not yet measured the impact of the campaign
Destination NSW employed a range of channels in the deployment of the Love NSW campaign, with mixed results. Against clear targets established for reach and engagement with target markets, the campaign met its target in eight of the 13 output measures. In connection with outcome measures, the campaign was unable to meet the majority of the targets established, including measures to 'take action' or to have an 'intention to take action'.
Destination NSW took a different approach to measuring the impact of the campaign, with objectives being linked to actual spending in regional New South Wales. Specifically, the primary impact objective of the campaign was to 'encourage dispersal of visitation and spend into smaller rural towns and businesses within NSW and Sydney'.
In its target setting, the impact objectives related to overnight visitor stays, visitor spend per night and average length of stay. At the time of this audit, Destination NSW did not have data against these measures, making it impossible to assess the effectiveness of this campaign.
|Partially achieved objective
|The Love NSW campaign met 60 per cent of the targets established for it in connection with measures of reach and audience engagement. The results varied depending on the elements of the campaign.
Partially achieved objective
|The Love NSW campaign met targets related to social sentiment for regional New South Wales and an uplift of appeal for travel in regional New South Wales. However, it failed to meet its targets against the more substantial measures of influence on intention to travel to regional New South Wales and linkage between the campaign and the travel to regional New South Wales.
|Destination NSW has not measured performance against the impact measures established for the Love NSW campaign.
The Road Trips campaign had not completed its evaluation at the time of the review, but for completed measures, it failed to meet a majority of targets established
The Road Trips campaign was a largely digital-focussed campaign, and based on the draft campaign evaluation report provided to the audit team, it partially met its impact targets. It met four of nine impact measures, partially met one, did not meet three and had not obtained data in connection with the final impact objective measure.
It is impossible to comment on the outcomes of the campaign as Destination NSW had not established targets for five outcome measures, and had not provided actual results against three measures. Of the remaining four measures, the Road Trips campaign failed to achieve three of the targets and exceeded one target.
Destination NSW established measures for overall visitor nights, spending, length of stay and market share for road trips as four of six impact objectives for the Road Trips campaign. At the time of this audit, Destination NSW did not have data to support measurement of these objectives. It is also notable that these broad measures are difficult to directly attribute to the campaign. For other impact measures, Destination NSW had neither established targets nor collected actual data. On this basis, it is not possible to comment on the impact of the Road Trips campaign.
|Partially achieved objective
|The Road Trips campaign met 44 per cent of the targets established for it in connection with measures of digital impressions delivered by the campaign.
|There was either no target or no actual result provided for 66 per cent of outcome measures. Of the remainder, the Road Trips campaign failed to meet the established targets for three of the four outcome measures.
|Destination NSW has not measured performance against the impact measures established for the Road Trips campaign.
3.2 Campaign economy
The absence of any cost-benefit analysis for these campaigns means the decision to invest in them was made without a clear view of potential value for money
The three advertising campaigns within the scope of this performance audit all cost in excess of $1.0 million. As noted in the introduction, a campaign over $1.0 million is required to undergo a cost-benefit analysis. While the urgent campaign provisions allow a cost-benefit analysis to be completed after the commencement of a campaign, Destination NSW did not complete cost-benefit analyses for any of the in-scope advertising campaigns by the time the campaigns ended.
Given the level of expenditure on these campaigns, it is appropriate that robust analysis underpins the decision-making of government on whether to invest in advertising campaigns. Further, when robust thinking of costs and benefits is undertaken at the commencement of an advertising campaign, this generally assists in robust evaluation after the completion of campaigns. Not having a cost-benefit analysis means that the investment was made without a clear view of the potential benefit-cost ratio, which would demonstrate that the campaign posed good value for money.
This approach was taken with full visibility of DCS, and given the unprecedented circumstances of the COVID-19 pandemic and the urgent nature of the advertising campaigns, the audit team understands the rationale for this decision.
The ‘Sydney - Love It Like You Mean It’ campaign made a negative economic contribution to New South Wales, and recorded only a marginal return on its investment
Of the two Love Sydney campaigns, Destination NSW was only required to undertake a cost-benefit analysis for the ‘Sydney - Love It Like You Mean It’ campaign, as it is the only one that incurred expenditure over $1.0 million.
The cost-benefit analysis reported a benefit to cost ratio of 0.01:1, which is considerably less than the Treasury expectation of a minimum 1:1 benefit to cost ratio. This translated to a negative net present value of the economic contribution from the campaign of -$2.79 million. As the campaign did not attract new economic activity from outside of the State (due to its focus on visitor activity by Sydney residents) the cost-benefit analysis methodology attributes minimal economic benefit from the campaign. This inevitably impacted the reported benefit to cost ratio.
As part of the same analysis, Destination NSW measures a ‘return on investment’ from the campaign (being the net present value of economic benefits less campaign costs), using a methodology which is unique for Destination NSW. Destination NSW advised that the methodology is consistent with similar methodologies used to assess advertising campaign in other tourism promoters around the country. The study assessed the return on investment from the ‘Sydney - Love It Like You Mean It’ campaign as being 1.3:1, which is considerably lower than the comparator identified in the campaign evaluation, which was 11:1.
The Love NSW campaign made a positive economic contribution
Using a methodology consistent with NSW Treasury's 'Cost-Benefit Analysis Framework for Government Advertising and Information Campaigns', Destination NSW determined that the ratio of benefits to costs from the Love NSW campaign was 12.5:1. In other words, the economic benefit arising from the campaign was 12.5 times its cost. This compares favourably to NSW Treasury's threshold of a 1:1 ratio. The economic benefit assessed in this cost-benefit analysis arises from an estimate of overnight bookings and day trips in regional New South Wales that would have been generated from the advertising campaign. This is an improvement on the benefit to cost ratio from the 2019–20 first phase of this campaign (which was 11:1).
As part of the same analysis, Destination NSW measured a ‘return on investment’ from the campaign (being the net present value of economic benefits less campaign costs), using a methodology which is unique for Destination NSW. Destination NSW advised that the methodology is consistent with similar methodologies used to assess advertising campaign in other tourism promoters around the country. The study assessed the return on investment from the Love NSW campaign as being $13.4 million against a cost of $1.7 million.
Destination NSW has not assessed the benefit cost ratio or return on investment from the Road Trips campaign
At the time of this audit, Destination NSW had not completed a cost-benefit analysis for the Road Trips campaign. Consequently, the audit has not been able to evaluate whether the Road Trips campaign was economical.
Destination NSW actively monitored data on the outputs and reach of the campaigns and responded with placement decisions to optimise the outputs from the campaigns
For all campaigns, Destination NSW received daily, weekly and monthly feeds of data and information about the effectiveness of reach and reaction to the campaigns. This allowed the organisation to make changes in the campaign’s media placement to optimise the outputs from the campaign, and thereby contribute to maximising value from the campaign.
3.3 Campaign efficiency
The creative expenditures for all in-scope campaigns were undertaken consistent with NSW Government requirements to support value for money procurement
Agencies are required to obtain three quotes when procuring a creative agency on the relevant pre-qualification scheme if the estimated cost of an individual expenditure in the creative content is greater than $150,000.
By virtue of re-using content, and coordinating its own creative content, creative, production and planning, all individual expenses in creative production was lower than the $150,000 threshold for all but one of the campaigns:
- Sydney - Love It Like You Mean It: total of $64,300
- Get Your Sydney On: total of $189,102 (with the largest single expenditure being for $128,381 for content production)
- Love NSW: total of $210,972.40 (with the largest single expenditure being for $67,463 for a media levy).
For the Road Trips campaign, which was a bespoke campaign, the production and research expenditure was $1,696,675. The largest component of this was a single expenditure of $649,000. Destination NSW undertook a competitive process drawing on the NSW Government Advertising Production Panel assessing three providers. The process undertaken and the criteria considered in that selection were reasonable to support a value for money selection.
Changes to media placement due to lockdowns and media re-direction was all achieved without any loss of purchased placements, albeit sometimes in lower value placements
The nature of the campaigns was to prompt visitor economy activity in the aftermath of the pandemic and as lockdowns were lifted.
This meant that the Love Sydney and Love NSW campaigns had a limited lead time to be placed in the market. That led to lower efficiency in media placement than would be considered normal, but was a consequence of the purpose of the campaign to be quick to market.
Given the unpredictable nature of the pandemic, numerous changes in campaign placement were required to react to changes in circumstances. This included pausing campaigns in some instances and/or re-directing to different regions or populations. Destination NSW monitored this actively and reacted swiftly with its media partner to make the decisions on media placement.
Inevitably this required late cancellations of media placements and short notice placements of new media bookings. Destination NSW's media partner has advised that all media placements were re-made, and no expenditure was lost. However, in some instances, the replacement resulted in lower quality placement.
© Copyright reserved by the Audit Office of New South Wales. All rights reserved. No part of this publication may be reproduced without prior consent of the Audit Office of New South Wales. The Audit Office does not accept responsibility for loss or damage suffered by any person acting on or refraining from action as a result of any of this material.
Parliamentary reference - Report number #360 - released (23 December 2021).