Assessment of the use of a training program

Overview

The Department of Finance, Services and Innovation (DFSI) and Service NSW's use of Franklin Covey's '7 Habits' program (the Program) met identified business needs according to a report released today by the Auditor-General for New South Wales Margaret Crawford. 

1. Introduction

This audit assesses the effectiveness and economy of the Department of Finance, Services and Innovation's, including Service NSW's, use of the Franklin Covey ‘7 Habits’ program (the Program). On 15 March 2018, the Hon. Victor Dominello MP, Minister for Finance, Services and Property, requested the Auditor General conduct this audit under section 27(B)(3)(c) of the Public Finance and Audit Act 1983 (the Act).

About the agencies

The Department of Finance, Services and Innovation (the Department) is the lead agency of the Finance, Services and Innovation cluster. The Department has a number of divisions and business units, including: ICT and Digital Government, Property and Advisory Group, Better Regulation, NSW Fair Trading, Government and Corporate Services, and Revenue NSW. At 30 June 2017, the Department (excluding Service NSW) had 5,239 full-time equivalent staff.

Service NSW is a central point of contact for customers accessing NSW Government Services. It is a Division of the Finance, Services and Innovation cluster and operates as an executive agency. As an executive agency, Service NSW is led by a Chief Executive Officer, who is responsible to the Minister for Finance, Services and Property but appointed by the Secretary of the Department of Finance, Services and Innovation. Service NSW was established in 2013 and has operated under the Finance, Services and Innovation cluster since July 2015. At 30 June 2017, Service NSW had 1,989 full-time equivalent staff.

About the Program

The Program that the Department and Service NSW are implementing, and which is the subject of this audit, is a professional development training course which focusses on organisational culture emphasising personal effectiveness, leadership development and change management. All staff in the Department and Service NSW will receive the training, which involves:

  • a 360-degree assessment where every staff member receives feedback from their manager, direct reports, and peers
  • a two-day training workshop, which will be delivered face to face by accredited facilitators
  • 2 years of online access to all training materials created by the provider of the Program.

As part of the licensing arrangement purchased by the agencies, the Program also provides access (at no extra cost) to the full range of the provider's training and development courses that might be useful for other learning and development activities. This includes courses to improve staff capability in communication skills, leadership, productivity and customer engagement. The Department is considering using one of these courses to develop leadership capabilities. Service NSW has integrated three of these courses into its people development curriculum.

Service NSW commenced the first sessions of the Program in May 2017. At 24 April 2018, around 1,000 staff had undertaken the training. Service NSW expects all staff to complete the Program by June 2019.

The Department of Finance, Services and Innovation commenced the first sessions of the Program in August 2017. At 18 April 2018, around 175 staff had undertaken the training. The Department expects all staff to complete the Program by December 2019.

Audit objective and criteria

The audit sought to assess the effectiveness and economy of the Finance, Services and Innovation cluster’s use of the Program. In making this assessment, we considered whether:

  1.  the Program is being used effectively, including whether
    1. there is an identified need for the Program
    2. the use of the Program meets the identified need
    3. Finance, Services and Innovation cluster agencies evaluate the effectiveness of the Program
  2. the Program is economical, including whether:
    1. the procurement complies with all relevant policies and processes
    2. funding and resources allocated to the Program are reasonable.
Conclusion
The Department of Finance, Services and Innovation, and Service NSW developed workforce strategies which identified a business need to improve organisational culture and staff engagement. The Program met the identified business needs and both agencies negotiated value for money contracts for the delivery of the Program when compared to other available options for training all staff.
However, the agencies did not document evidence to show that training all staff members was necessary to meet their business needs, as compared with training fewer staff members at a lower overall cost. As a result, we are unable to form a view on whether the approach to train all staff members was economical. The agency heads have subsequently provided information supporting their decisions to train all staff members. This information indicates their decisions were based on evidence that this would meet the goals of their workforce strategies, including improving employee engagement scores and organisational culture change.
The Department is paying $1,320,700, over three years, for up to 5,600 staff to participate in the Program ($235.84 per person). Service NSW is paying $595,000, over two years, for up to 2,400 staff to participate in the Program ($247.92 per person).
The agencies are collecting the data they need to evaluate the Program and there is some evidence that the Program is achieving its objectives in Service NSW. Due to the timing of this audit, there is not yet enough information available to comment on whether the Program is achieving its objectives in the Department.

Sector-wide learnings

Implementing robust learning and development frameworks

  1. Agencies should evidence decisions about how proposed learning and development opportunities will meet staff and business needs - both in the program design, and through evaluation. In many cases, organisations may have unique needs or circumstances, or may want to trial innovative approaches to improving organisational capability. Innovation should be encouraged, to avoid the risk that agencies are locked into outdated training and development models. However such approaches should be balanced by ensuring that business needs are well scoped and defined.
     
  2. Agencies implementing innovative or new approaches to learning and development should build-in iterative evaluations (such as pulse surveys, or collecting post-participation qualitative feedback) to ensure that the training is delivered on intended benefits, and to inform improvements to ongoing rollout.
     
  3. Agencies implementing innovative or new training programs should ensure they build enough flexibility into contracts so that they can assess how well programs are meeting staff and business needs, and use evidence to inform whether further rollout should occur.

Recommendations

Section two of this report provides further information on the issues that these recommendations seek to address.

  1. We recommend that the Department of Finance, Services and Innovation:
    1.  improve the guidance provided to NSW Government agencies engaging in direct procurement negotiations
    2. establish specific targets in its learning and development evaluation framework and measure how effectively the Program is assisting it to achieve these targets.
  2. We recommend that the Department of Finance, Services and Innovation, and Service NSW:
    1. conduct post-completion reviews of the Program procurement processes in line with ICAC’s ‘Guidelines for managing risk in direct negotiations’
    2. assess the results of their 2018 People Matter Employment Surveys to evaluate whether the Program is achieving intended business benefits, including culture change. This evaluation should inform a decision on whether implementation should continue.

2. Assessment of the use of a training program

2.1 How the Program meets business needs

The Department and Service NSW used evidence-based workplace strategies to identify the need for the Program

Service NSW

In 2016, Service NSW began implementing an organisation-wide strategic plan that included the goal of creating a world class team that attracts and retains the best talent and nurtures their workplace culture. To do this, Service NSW began developing a People Development Strategy in 2016. The development of the strategy was informed by:

  • results and analysis of the staff surveys (including Pulse and People Matters Engagement Surveys) which demonstrated that staff engagement was declining across Service NSW (from 2015 through to 2017)
  • feedback from staff consultation sessions
  • feedback from a 2017 organisational strategy review.

In March 2017, Service NSW's Executive Leadership Team endorsed the People Development Strategy. The strategy identified the need for a program that focussed on staff capability and was accessible for all staff. Other requirements identified as part of the strategy included:

  • the ability of the Program to maintain (or improve) high levels of customer satisfaction and employee engagement
  • the Program would grow leadership capabilities to support future workplace needs
  • the development of a common language that aligned with the organisational culture.

Department of Finance, Services and Innovation

In 2016, the Department identified the need to improve employee engagement, the management of change and productivity. As a result, the Secretary approved the People Essentials strategy in December 2016. Its purpose was to:

  • build specific capabilities across the Department, including:
    • authentic/adult conversations
    • goal and objective setting
    • resilience
    • time and priority management
  • build a positive and productive workplace
  • ensure the Department is maximising the potential of its staff
  • support and facilitate cultural transformation across DFSI.

The Secretary and the Department Executive agreed that the ‘People Essentials’ program should be facilitated through a partnership model with an external agency.

Agencies did not document evidence to support their decision to train all staff

The Department and Service NSW identified a whole-of-organisation training approach as necessary to meet a business need in their respective workforce strategies. Subsequently, the agency heads provided additional information to the Audit Office to support their decisions to train all staff members. This information includes evidence that training all staff would meet the goals of their workforce strategies, including improving employee engagement scores and organisational culture change.

Targeted training programs that focus on leaders rather than training all staff, can also effect cultural change in organisations. Targeted training programs, because they involve training fewer staff, have the potential to be cheaper than whole-of-organisation training approaches.

While the Department and Service NSW invest in targeted training programs to increase management and leadership capabilities, they advised that these approaches are slow to roll out and more expensive on a per-head basis to run. The agencies advised that, in their professional judgement, a targeted training approach would not be effective in meeting their needs, and may have a negative impact on staff engagement.

The agencies did not document evidence that training all staff members was necessary to achieve cultural change, as compared with training fewer staff at a lower overall cost. As a result, we are unable to form a view on whether the approach to train all staff members was the most economical.

The Program met the identified needs of the agencies

The Department and Service NSW assessed the Program against their respective business needs. The agencies collected information regarding the Program to provide reasonable assurance that it would meet their needs. Both organisations conducted analysis of other whole-of-organisation training options, including other potential training courses, as part of the procurement process.

Service NSW

Service NSW considered whether other potential solutions:

  • could be delivered outside the Sydney metropolitan area
  • aligned with Service NSW's organisational culture
  • could be delivered by internal facilitators
  • were appropriate for staff at all levels
  • met the strategic needs expressed in the People Development Strategy.

Service NSW considered developing an annual program using a series of providers, but decided that a multiple vendor solution would be less likely to build a consistent Service NSW culture. Service NSW also noted that using multiple vendors would likely require the use of courses only provided in Sydney.

Service NSW concluded that the Program was the only option that met all their business needs.

Department of Finance, Services and Innovation

The Department conducted a comparison of seven programs, including indicative costs, which were identified as having possible alignment to the needs of its People Essentials strategy.

The Department assessed the options against the following criteria:

  • met capability development areas as stated by the Secretary
  • quality of content, materials and reputation
  • price/value for money
  • target deliverables and/or outcomes
  • flexibility of delivery options.

An example of some of the analysis undertaken by DFSI has been reproduced in Exhibit 1.

The Department concluded that the Program most closely aligned with its identified business needs.

Exhibit 1: Analysis conducted by the Department of Finance, Services and Innovation on its staff training needs against the content of training programs
Identified staff training needs Training program 11 Training program 21 Training program 31 Training program 41 Training program 51 Training program 61 Selected training program
Authentic/Adult Conversations - check_circle_green_0.png - - check_circle_green_0.png check_circle_green_0.png check_circle_green_0.png
Goal/Objective Setting - - - check_circle_green_0.png - check_circle_green_0.png check_circle_green_0.png
Resilience check_circle_green_0.png - check_circle_green_0.png - - - check_circle_green_0.png
Time & Priority Management - - - - - check_circle_green_0.png check_circle_green_0.png
Expectations - check_circle_green_0.png - - check_circle_green_0.png - check_circle_green_0.png
Positive & Productive Workplace check_circle_green_0.png - check_circle_green_0.png check_circle_green_0.png check_circle_green_0.png check_circle_green_0.png check_circle_green_0.png

1 Other training programs considered by Department of Finance, Services and Innovation. Names of training programs withheld.

The Department has not set specific targets to assess the success of the Program

The Department and Service NSW have both developed evaluation frameworks to measure the effectiveness of the Program. The evaluation methodologies describe a set of expected outcomes and detail how the agencies would progress against those outcomes (see Appendix two). The evaluation methodologies, if implemented fully, will effectively measure the impact of the training program across the organisations. Both the Department and Service NSW are collecting data, in line with the evaluation frameworks, to effectively measure outcomes.

Service NSW has established specific targets for employee engagement as part of its strategic plan, and uses the targets to assess whether the Program is successful. The Department has not set specific targets for its outcome measures. Without specific target measures, the Department cannot assess whether the Program has been fully effective.

In Service NSW, initial evaluation data indicates that the Program is effective

Service NSW and the Department ask training participants to provide feedback on the Program. Participants speak favourably about the Program and there is some evidence that the Program is achieving its objectives in Service NSW. However, it is too early to comment on the overall effectiveness of the Program in the Department.

The Department and Service NSW collect data on staff engagement from two main sources:

  • People Matter Employee Surveys – coordinated by the Public Service Commission for all NSW public sector employees
  • Pulse Surveys – internally coordinated and offered to the relevant agency staff.

The 2018 Pulse results for Service NSW indicate a six per cent increase in overall employee engagement from 69 per cent in 2017 to 75 per cent in 2018.

Additionally, Service NSW provided case study information on the impact of the Program on two regional offices. The case studies provided additional evidence that the Program is responsible for improving staff engagement and effectiveness.

Due to the timing of this audit, there is not yet enough information available to comment on whether the Program is achieving its objectives in the Department.

2.2 Economy of the program

The agencies generally complied with the NSW Procurement Policy Framework and their internal procurement frameworks and financial delegations, although we identified some areas with Service NSW's procurement approach that could be more robust

The NSW Procurement Policy Framework for NSW Government Agencies (July 2015) details mandatory requirements for agencies that are procuring via direct negotiation. It also refers agencies to guidance provided in the ICAC 'Guidelines for managing risks in direct negotiations’. The Department complied with all Framework requirements. Service NSW generally complied with the Framework requirements, however we found minor issues with the timing of the market analysis and quality of the risk assessment. These issues were unlikely to have had an impact on the outcome of the procurement.

The Department and Service NSW did not document their consideration of guidance provided by the ICAC Guidelines. This is covered in more detail below.

Exhibit 2: Compliance with NSW Procurement Policy Framework - Complex market procurement method requirements
  SNSW DFSI

Ensure that the procurement strategy addresses in detail the justification and suitability of the selected procurement method. In the case of direct negotiations/sole sourcing arrangements, a comprehensive analysis of the market and all relevant factors should be undertaken to demonstrate that a competitive process does not need to be conducted

(add icon) (add icon)
Conduct a risk assessment for the procurement, including an assessment and treatment of procurement process risk arising from the use of the procurement method (add icon) (add icon)
Follow internal delegations process, which will ordinarily involve obtaining the approval of the agency’s Chief Procurement Officer, and in some instances also require the approval of a more senior person in the agency, depending on the nature or value of the procurement  (add icon) (add icon)
Key (add icon) Requirements met  (add icon) Requirements met, minor issues noted

Source: NSW Procurement Policy Framework for NSW Government agencies July 2015, Audit Office research.

Service NSW

Service NSW staff, on the basis of experience and expertise, identified the Program as a solution that could meet business needs and initiated the procurement approval process. Service NSW staff should have documented their market assessment at this preliminary stage. During the procurement approval process, the Chief Procurement Officer identified this deficiency and required supporting documentation. Service NSW staff then documented a review of three well known organisational development programs and considered developing a fourth option to build an annual program using a series of professional development vendors. Service NSW completed the market assessment before the Chief Executive Officer approved the procurement in line with the NSW Procurement Policy Framework (July 2015).

Service NSW did not conduct a detailed risk assessment for the procurement and relied on the engagement of internal procurement staff to manage the risks associated with direct negotiation.

The Department of Finance, Services and Innovation

The Department conducted research on a range of training programs which might meet the identified business needs. It also collected information on other training and development programs delivered, or being considered for delivery, by divisions within the Department. The Department's decision to procure via direct negotiation was informed by this analysis. As discussed above, both agencies concluded that the Program was the only option that met their business needs. The agencies applied a reasonable process to support this conclusion.

The Department conducted a detailed risk assessment of the procurement process and used the expertise of their procurement staff to address the risks associated with the direct negotiation. The Deputy Secretary, Government Corporate Services, signed the contract in accordance with the Department’s delegations.

The procurement guidance for direct negotiations should be improved

The NSW Procurement Policy Framework refers agencies to guidance provided in the ICAC 'Guidelines for managing risks in direct negotiations'. The ICAC Guidelines include general principles to inform any decision to enter into direct negotiations. They are not intended to specifically address the circumstances for every type of procurement.
As a result, there is a lack of clarity regarding what processes or checks should be applied to different types of direct negotiations. This contrasts with general government procurement, where agencies use expenditure thresholds to determine the best way to achieve value for money.

The Department and Service NSW did not document how they considered the ICAC Guidelines as part of their procurement process. As part of this audit, the agencies provided additional information.

Exhibit 3 describes our assessment of the consistency of the agencies' procurements with the ICAC guidelines. Of the items in the list that were not met, the guidelines are not clear on whether some would have been appropriate in these circumstances.

For example, the creation of a procurement specific probity plan may be more appropriate for high dollar-value infrastructure procurements.

The agencies advised that they do not believe that all elements of the checklist were relevant for their procurements. Their rationale for this understanding is included in Exhibit 3.

Exhibit 3: Compliance with the Independent Commission Against Corruption’s (ICAC) checklist for undertaking direct negotiations
Checklist for undertaking direct negotiations SNSW DFSI Agency comment
Decide whether direct negotiations can be justified

check_circle_green_1.png

 

check_circle_green_1.png N/A
Verify that the project or proposal is consistent with the agency’s overall strategic plan check_circle_green_1.png check_circle_green_1.png N/A
Check that there is an appropriate project or business case and that funding has been approved check_circle_green_1.png check_circle_green_1.png N/A
Ensure that the decision to enter into direct negotiations has been made or approved at a senior level within the agency check_circle_green_1.png check_circle_green_1.png N/A
Ensure that the decision to enter into direct negotiations, and the reasons for it, are recorded and are publicly accessible yellow_minus_0.png yellow_minus_0.png This information is publicly accessible though the Government Information (Public Access) Act 2009. The contract details are publicly accessible on the eTender website.
Perform cost-benefit analysis or similar study to verify that direct negotiations will not sacrifice value for money yellow_minus_0.png yellow_minus_0.png A desktop review of the price of other training programs was conducted. The cost of the Program was also considered against the price advertised to the public.
Obtain conflict of interest declarations from persons associated with the project including staff, the proponent’s employees and advisers yellow_minus_0.png yellow_minus_0.png All senior executives are required to submit an annual conflict of interest declaration. Additionally, staff are required to declare any conflicts of interest under the staff code of conduct. No conflicts were identified or disclosed.
Ensure that the contract entered into gives the agency the authority to intervene in the event of poor performance or unethical conduct check_circle_green_1.png check_circle_green_1.png N/A
Engage external assistance as necessary check_circle_green_1.png check_circle_green_1.png N/A
Prepare a probity plan yellow_minus_0.png yellow_minus_0.png This was not deemed necessary, from a risk or financial perspective, for an off the shelf product, from a well established provider, and for a relatively low monetary value.
Establish an agreed negotiation protocol check_circle_green_1.png check_circle_green_1.png N/A
Ensure that an auditable document trail is established check_circle_green_1.png check_circle_green_1.png N/A
Obtain an independent estimate of the price or financial consideration yellow_minus_0.png yellow_minus_0.png This was not deemed necessary given the product is an off the shelf program, readily available to the public.
Establish project benchmarks for deliverable items check_circle_green_1.png check_circle_green_1.png N/A
Arrange for an ‘open-book’ relationship with the proponent check_circle_green_1.png check_circle_green_1.png N/A
Segregate duties check_circle_green_1.png check_circle_green_1.png N/A
Form a steering committee or similar oversight mechanism check_circle_green_1.png check_circle_green_1.png N/A
Supervise the project to ensure that the contract is adhered to check_circle_green_1.png check_circle_green_1.png N/A
Conduct a post-completion evaluation of the project to assess whether value for money has been obtained. yellow_minus_0.png yellow_minus_0.png This will be undertaken once the critical mass of participants have undertaken the Program.
Key    check_circle_green_1.png ICAC checklist item met    yellow_minus_0.png ICAC checklist item not met

Source: ICAC Guidelines for managing risks in direct negotiations and Audit Office analysis.

Neither agency published information on their justification for direct negotiation, or obtained procurement specific conflict of interest declarations

By their nature, direct negotiations lack transparency and the benefit of competition in the procurement process. Equally important, direct negotiations risk the perception of conflict of interest or improper conduct.

The ICAC Guidelines describe the need for transparency in direct negotiations:

To avoid suggestions of impropriety, the agency should record the reasons it has chosen to enter into direct negotiations. In order to maintain transparency, it should, where appropriate, make the decision and supporting reasons public, possibly by placing a notice on the agency website.

Both agencies documented the reasons for their decisions to enter into direct negotiations but did not publish this information or make it publicly accessible. While the agencies' advised that information is publicly accessible though the Government Information (Public Access) Act 2009, the Act requires members of the public to apply for relevant documents and pay a fee. This is not consistent with the ICAC Guideline suggestion that it be more readily available.

The ICAC Guidelines also describe the need for conflict of interest declarations:

The agency needs to be confident that there are no financial or personal associations between its staff and the proponent that could influence decisions made by public officials. To lower the chances of negotiations being driven by private interests, all individuals party to the project should be required to sign a declaration stating that they are not aware of any conflicts of interest. This requirement should extend to staff working for the proponent, external advisors and internal staff.

Neither agency required staff involved with the procurement, or the Program provider, to provide procurement specific conflict of interest declarations. Both agencies advised that all senior executive staff provide annual conflict of interest disclosures, and the agencies require senior executive staff to update their disclosures when circumstances change throughout the year. The Department and Service NSW also have codes of conduct which require staff to declare conflicts of interest. No conflicts of interest were identified or disclosed.

It is our view that, in line with the ICAC Guidelines, agencies should require all staff involved in direct negotiations to provide procurement-specific conflict of interest declarations.

Funding and resources allocated to the Program are reasonable

In 2016–17, the Department spent around $6.0 million on training. The value of the Department's contract for the Program is $1.3 million over three years. In 2016–17, Service NSW spent around $1.1 million on training. The value of Service NSW's contract for the Program is $595,000 over two years.

For the Department and Service NSW, the ratio of staff training expenses to total employee expenditure is comparable to other NSW Government agencies of comparable size and function (refer to Exhibit 4).

Exhibit 4: Comparison of NSW Government Department’s spend on training in 2016–17
  Employee related
expenses
2016–17
Training
expenses
2016–17
Training
expenditure/employee
related expenses
  $'000 $'000 (%)
Agency X* 503,196 3,018 0.60
Service NSW 159,767 1,099** 0.69
Department of Finance, Services and Innovation 614,510 6,179 1.01
Agency Z* 1,550,708 20,038 1.29
Agency A* 130,449 1,691 1.30
Agency Y* 84,060 1,491 1.77
* Data from other NSW Government Departments is provided for comparison only, as such Department names have been withheld. Only Departments with comparable data have been presented.
** Source: Service NSW (unaudited).
Source: Various NSW Government Departments audited financial statements 2016–17.

The Department and Service NSW selected a cost-effective facilitation option

Both agencies considered the option of internal and external facilitation of the Program. Internal facilitation requires Department and Service NSW staff to undertake Program accreditation so they can teach the Program. External facilitation requires agencies to pay an additional fee for the Program provider to deliver the training.

Both agencies chose the internal facilitation option. This option provided agencies with flexibility to deliver training across the state in a cost-effective manner while developing staff capability.

The Department and Service NSW negotiated value for money contracts with the Program provider

The Department of Finance, Services and Innovation negotiated a price with the training provider that equates to $236 per participant for delivery to all staff. Service NSW negotiated a price of $248 per participant for delivery to all staff (see Exhibit 5). The cost per participant for the Program for both agencies is significantly lower than the price of the Program advertised to the public. The training provider currently charges members of the public $1,495 for the two day in-person training program.

We compared the Program syllabus and negotiated price for the two agencies with similar competency based training programs available to government. Single day externally facilitated programs that address some, but not all, elements of the Program syllabus cost around $700 per participant.

Exhibit 5: Cost per participant excluding GST (includes direct costs paid to training provider only)
  DFSI SNSW
All Access Pass Plus (training licence)* $991,200 $595,000
Participation kits $280,000 -**
Other amounts associated with program delivery $49,500 -**
Total value of contract $1,320,700 $595,000
Participants 5,600 2,400
Cost per participant $235.84 $247.92

Note: These costs represent the training providers fee and the cost of training materials. Variations to the number of participants would attract additional cost. These figures do not include other direct and indirect costs of training such as venue hire, catering, administrative costs and the staffing resources required to roll out the Program.
* Training licence gives training participants access to several of the training provider's programs for 24 months from the commencement of training.
** Items provided as complimentary.
Source: Department of Finance, Services and Innovation and Service NSW.