Engagement of probity advisers and probity auditors

Overview

Three key agencies are not fully complying with the NSW Procurement Board’s Direction for engaging probity practitioners, according to a report released today by the Acting Auditor-General for New South Wales, Ian Goodwin. They also do not have effective processes to achieve compliance or assure that probity engagements achieved value for money.

Executive summary

Probity is defined as the quality of having strong moral principles, honesty and decency. Probity is important for NSW Government agencies as it helps ensure decisions are made with integrity, fairness and accountability, while attaining value for money.

Probity advisers provide guidance on issues concerning integrity, fairness and accountability that may arise throughout asset procurement and disposal processes. Probity auditors verify that agencies' processes are consistent with government laws and legislation, guidelines and best practice principles. 

According to the NSW State Infrastructure Strategy 2018-2038, New South Wales has more infrastructure projects underway than any state or territory in Australia. The scale of the spend on procuring and constructing new public transport networks, roads, schools and hospitals, the complexity of these projects and public scrutiny of aspects of their delivery has increased the focus on probity in the public sector. 

A Procurement Board Direction, 'PBD-2013-05 Engagement of probity advisers and probity auditors' (the Direction), sets out the requirements for NSW Government agencies' use and engagement of probity practitioners. It confirms agencies should routinely take into account probity considerations in their procurement. The Direction also specifies that NSW Government agencies can use probity advisers and probity auditors (probity practitioners) when making decisions on procuring and disposing of assets, but that agencies:

  • should use external probity practitioners as the exception rather than the rule
  • should not use external probity practitioners as an 'insurance policy'
  • must be accountable for decisions made
  • cannot substitute the use of probity practitioners for good management practices
  • not engage the same probity practitioner on an ongoing basis, and ensure the relationship remains robustly independent. 

The scale of probity spend may be small in the context of the NSW Government's spend on projects. However, government agencies remain responsible for probity considerations whether they engage external probity practitioners or not.

The audit assessed whether Transport for NSW, the Department of Education and the Ministry of Health:

  • complied with the requirements of ‘PBD-2013-05 Engagement of Probity Advisers and Probity Auditors’
  • effectively ensured they achieved value for money when they used probity practitioners.

These entities are referred to as 'participating agencies' in this report.

We also surveyed 40 NSW Government agencies with the largest total expenditures (top 40 agencies) to get a cross sector view of their use of probity practitioners. These agencies are listed in Appendix two.

Conclusion

We found instances where each of the three participating agencies had not fully complied with the requirements of the NSW Procurement Board Direction ‘PBD-2013-05 Engagement of Probity Advisers and Probity Auditors’ when they engaged probity practitioners. We also found they did not have effective processes to achieve compliance or assure the engagements achieved value for money.

In the sample of engagements we selected, we found instances where the participating agencies did not always:

  • document detailed terms of reference
  • ensure the practitioner was sufficiently independent
  • manage probity practitioners' independence and conflict of interest issues transparently
  • provide practitioners with full access to records, people and meetings
  • establish independent reporting lines   reporting was limited to project managers
  • evaluate whether value for money was achieved.

We also found:

  • agencies tend to rely on only a limited number of probity service providers, sometimes using them on a continuous basis, which may threaten the actual or perceived independence of probity practitioners
  • the NSW Procurement Board does not effectively monitor agencies' compliance with the Direction's requirements. Our enquiries revealed that the Board has not asked any agency to report on its use of probity practitioners since the Direction's inception in 2013. 

1. Recommendations

  1. Participating agencies should review and revise probity policies, processes and systems to ensure they:
    • comply with the Direction's requirements and:
      • enhance internal probity employees' independence and capability
      • justify the use of external probity practitioners
      • ensure selected suppliers have proven experience in an operational field relevant to the nature and risks of the specific projects
      • document detailed terms of reference, which clearly and comprehensively articulate the principles of probity, the scope of work, how conflicts of interest and disputes will be dealt with, establish independent reporting lines, and enable access to records, meetings, premises and personnel
      • ensure agency heads are satisfied risks associated with probity practitioners' potential conflicts of interests and/or independence have been adequately mitigated before they are engaged
      • implement internal controls that effectively identify, mitigate, resolve and report breaches of the Direction's requirements
         
    • evaluate probity practitioner performance against an agreed scope to assess whether value for money has been achieved after an engagement
       
    • capture lessons learned from external practitioner engagements to build internal probity management capability and capacity
       
    • ensure internal audit functions, under the oversight of audit and risk committees, regularly review agency probity governance and processes as part of their cyclical audit programs.
       
  2. The NSW Procurement Board should support agencies more by developing and implementing criteria that ensure prequalified practitioners have the required capability and experience to deliver quality outcomes.
     
  3. The Department of Premier and Cabinet should work with Gateway Coordination Agencies to develop probity guidance that helps agencies consistently apply the Direction's requirements.

1. Engagement of probity advisers and probity auditors

There are no professional standards and capability requirements for probity practitioners

NSW Government agencies use probity practitioners to independently verify that their procurement and asset disposal processes are transparent, fair and accountable in the pursuit of value for money. 

Probity practitioners are not subject to regulations that require them to have professional qualifications, experience and capability. Government agencies in New South Wales have difficulty finding probity standards, regulations or best practice guides to reference, which may diminish the degree of reliance stakeholders can place on practitioners’ work.

The NSW Procurement Board provides direction for the use of probity practitioners

The NSW Procurement Board Direction 'PBD-2013-15 for engagement of probity advisers and probity auditors' outlines the requirements for agencies' use of probity practitioners in the New South Wales public sector. All NSW Government agencies, except local government, state owned corporations and universities, must comply with the Direction when engaging probity practitioners. This is illustrated in Exhibit 1 below.

The following diagram notes that use of probity advisers/auditors should be an exception rather than the rule and also highlights guidelines for when agencies use their internal probity resources and when agencies use external probity resources. For an accessible version email communications@audit.nsw.gov.au
Exhibit 1: Compliance requirements of the Direction

Participating agencies did not fully comply with the Procurement Board's Direction

We examined three participating agencies' compliance with the Direction's requirements over the period 1 July 2015 to 30 June 2018. 

We noted all three agencies include probity principles in their policies and procedures, and offer probity training to their staff. However, we found instances, in the sample of engagements we selected at each participating agency, where the agencies had not fully complied with other requirements of the Direction. When engaging probity practitioners, agencies were:

  • not documenting and applying the Direction's recommended engagement criteria before determining whether to engage a probity practitioner
  • not documenting or properly defining terms of reference
  • not evidencing how agency heads were satisfied there were no unmitigated conflicts of interest and/or independence issues with the selected probity practitioners
  • not considering or implementing sufficient safeguards to address actual or perceived conflicts of interest and/or independence issues.

We surveyed the top 40 NSW Government agencies' use of probity practitioners

We surveyed 40 NSW Government agencies with the largest expenditures in sector (top 40 agencies) to understand the use of probity practitioners across the New South Wales public sector. Exhibit 2 presents key observations from the survey results.

This diagram shows that agencies are predominately engaging the same 3 suppliers while 12 per cent or 5 agencies did some form of performance assessment after engagement. For an accessible version, email communications@audit.nsw.gov.au
Exhibit 2: Survey results of top 40 NSW Government agencies' use of probity practitioners

1.1 Things to consider before engaging probity practitioners

The Direction requires agencies to:

  • have effective internal mechanisms to ensure they consider probity principles throughout their decision making processes
  • use existing resources from within their agency, cluster agencies or elsewhere in the New South Wales public sector before engaging external probity practitioners
  • consider all criteria for determining whether to engage a probity practitioner
  • use external probity practitioners by exception rather than as a rule.

Probity principles are not defined

There are no prescribed probity principles in the New South Wales public sector. We identified five common probity principles in the participating agencies' internal policies and probity reports. These are detailed below in Exhibit 3: Probity principles most commonly used by agencies.

All three participating agencies have defined probity principles in their procurement policies and procedures. However, Transport for NSW was the only participating agency that incorporated probity considerations in its asset disposal policies and procedures.

The participating agencies employ inconsistent approaches to clearly setting out the probity principles they want probity practitioners to follow during their engagements. Inadequate terms of reference may have contributed to this. We found instances where Transport for NSW and the Department of Education allowed the probity practitioners to follow their own principles as articulated in their own probity reports. The Ministry of Health, however, had ensured probity principles were prescribed in the terms of reference of each engagement we looked at.

This diagram shows the commonly used principles for probity are value for money, impartiality, confidentiality, accountability and transparency and dealing with conflicts of interest. For an accessible version email communications@audit.nsw.gov.au
Exhibit 3: Probity principles most commonly used by agencies

Agencies' policies do not adequately ensure probity practitioner engagements are justified

The three participating agencies could not provide us with evidence to justify why they sought external probity advisory services for the engagements we looked at. 

Most of the top 40 agencies we surveyed indicated they internally assess projects against the Direction's criteria before deciding whether to engage external probity practitioners. However, eight agencies self declared their procurement policies do not clearly articulate when an external probity practitioner should be engaged.

The Direction requires agencies to use external probity practitioners by exception rather than as a rule. It makes it clear that agencies should not use probity practitioners as 'an insurance policy' to avoid accountability for decisions or as a substitute for good management practices.

The Direction requires agencies to use the following criteria to determine when they should engage a probity practitioner:

  • where the integrity of the process (or part of it) may be called into question
  • where the project is politically sensitive and/or potentially controversial
  • to avoid a perception of bias or favouritism
  • where the process is extremely complex
  • where substantial costs are involved in preparing submissions or substantial government funding is involved.

We reviewed the three participating agencies' policies and procedures to see if they include the Direction's criteria to determine when they should engage probity practitioners.

Transport for NSW’s goods and services procurement policy does include the Direction's engagement criteria. However, its infrastructure procurement policy requires a probity adviser to be engaged for all:

  • medium and major works contracts (above $10.0 million)
  • contracts for the supply of vehicles, vessels and rolling stock.

This approach fails to sufficiently consider the qualitative criteria in the Direction and may result in probity practitioners being engaged unnecessarily.

The Department of Education's policies or procedures do not clearly define the criteria it uses, while the Ministry of Health's policies and procedures reference probity to the Direction's requirements.

Agencies did not adequately document internal probity resource assessments

The Direction requires agencies to assess whether there are suitably qualified resources within their agency, cluster or elsewhere in the New South Wales public sector that could provide quality probity services before engaging an external probity practitioner.

For the engagements we reviewed, we found that the three participating agencies did not adequately document their resource requirement and availability assessments before engaging external probity practitioners. Conversely, 38 of the top 40 surveyed agencies reported that they did make these assessments.

Using internal resources may raise perceived or actual independence issues. These can be mitigated if agencies use their internal audit functions, under the oversight of their audit and risk Committees, to regularly review probity governance and processes as part of their cyclical audit program.

The Direction also requires staff responsible for asset procurement and disposal to be familiar with probity issues and to be trained in relevant NSW Government policies and procedures. 

We found all participating agencies offered internal probity training to employees responsible for procurement. However, 25 of the top 40 surveyed agencies reported that they had not given relevant staff probity specific training.

1.2 Selection of probity practitioners

The NSW Procurement Board has no specific qualification criteria for probity practitioners

The qualifications, work capabilities and experience required of probity practitioners are not clearly defined and the current vetting process gives no visibility of whether practitioners are suitably qualified for unique or complex projects. The NSW Government's Performance and Management Services Scheme (PMS scheme) offers a pool of prequalified suppliers to government agencies. Many agencies, including the Department of Education and Ministry of Health, rely on this Board’s prequalification scheme to select probity practitioners.

The Board is responsible for vetting suppliers to ensure they meet quality, safety and viability standards in the prequalification process. However, we found it only has high level evaluation criteria to assess the capability of probity practitioners. 

The Board should clearly define and assess the qualifications, work capabilities and experience required of probity practitioners to ensure the desired, quality outcomes are achieved. The vetting process should give visibility of practitioners' suitability for unique or complex projects.

1.3 Engagement terms of reference

Agencies did not clearly scope the probity engagements we reviewed

The Direction requires agencies to define and document the scope of probity engagements, and this should:

  • cover the intended role of the practitioner
  • clarify how conflicts of interest and disputes are to be managed
  • enable practitioners to access sufficient information in the procurement or sale process
  • outline how the practitioner can access additional information
  • give the practitioner full authority to access records, personnel, meetings and premises
  • detail reporting timelines and the expected completion date of the engagement
  • specify who the practitioner reports to
  • define ownership of the report (including working papers and supporting materials)
  • specify arrangements to secure materials and evidence during the engagement.

We found the three participating agencies did not clearly scope the probity engagements we looked at or document the terms of reference. Yet 39 of the 40 surveyed agencies reported they always agree and document the scope of their probity practitioner engagements. 

The Department of Education references the PMS scheme rules standard form of agreement in its terms and conditions for probity engagements. Transport for NSW references its standard terms and conditions for professional services and standing offer deed in its engagements with probity advisers. In our view, using these generic terms and conditions does not sufficiently meet the Direction's requirements. This is because the PMS scheme rules do not cover the intended role of probity practitioners and reporting lines, and Transport for NSW's standard terms do not sufficiently cover the authority probity practitioners require to access information and premises they need to fully perform their duties.

In some instances, we found the three participating agencies did not agree engagement terms of reference. Instead, they relied on the probity practitioner's work proposal. This can make it difficult for probity practitioners to access records and raise probity concerns to appropriate levels. 

Agencies should clearly scope probity engagements and develop detailed terms of reference for each engagement.

1.4 Managing conflicts of interest and independence issues

Agencies are not managing probity practitioner conflicts of interest and independence well

To ensure probity practitioners can fully perform their duties and meet stakeholder expectations, the Direction requires agency heads to be satisfied that probity practitioners are independent and free from real or perceived conflicts of interest. 

None of the participating agencies were able to demonstrate their agency head had sufficiently considered this aspect before engaging probity practitioners. In some instances, we found the participating agencies:

  • relied on the probity practitioners' self declaration that they were independent and had no conflicts of interests
  • did not obtain sufficient evidence to confirm the practitioners had safeguards in place to address such issues.

Where agencies use internal procurement managers or equivalents to perform probity adviser roles, such resources may, or may be perceived to be less likely to challenge internal processes and procedures, because they are generally not independent of the procurement process.

Such internal independence threats may be mitigated, to some extent, if agencies use their internal audit functions, under the oversight of their audit and risk Committees, to regularly review agency probity governance and related processes as part of their cyclical audit program.

The Direction also states that there is a general presumption against agencies engaging auditors already engaged in other work within the agency.

Engaging the same probity practitioner on an ongoing or serial basis over several related or unrelated issues increases self interest and familiarity risks that may threaten the actual or perceived independence of the practitioner.

All three participating agencies repeatedly used the same probity practitioners for several similar projects. The Department of Education and Ministry of Health engaged the same director and principal probity adviser across multiple projects. Transport for NSW approved the engagement of a probity adviser because the adviser was familiar with a supplier's work on different projects within the Transport cluster. 

Of the top 40 agencies, 35 provided us with data on their spend on probity services by provider. Over the last three financial years these agencies have, in terms of the value of the engagements, predominately engaged the same three external probity service providers. 16 of these agencies engaged the same provider continuously over a 12 month period.

The participating agencies believe the concentration of work allocated to these three firms is because the number of practitioners with suitable qualifications and expertise, to provide quality probity services for complex projects, is limited.

Exhibit 4: Percentage of probity engagements by provider over the last three financial years
Probity service provider: % Engagements by dollar value
Company A 46
Company B 28
Company C 9
Others 17

Source: Top 40 agency survey responses (unaudited).

1.5 Oversight and reporting on the use of probity practitioners

The NSW Procurement Board is not oversighting agencies' use of probity practitioners

The NSW Procurement Board is responsible for oversighting and monitoring agency compliance with the Direction. Since issuing the Direction in 2013, the Board has not monitored agency compliance or requested any agency to report on its use of probity practitioners.

The three participating agencies do not report data on their use of probity practitioners internally. This makes it difficult for those charged with governance to understand and monitor the quantity and quality of work performed by probity practitioners. Only one of the top 40 surveyed agencies reported that it periodically reports its use of probity practitioners to those charged with governance.

Three of the top 40 agencies, including the Department of Education, could not provide us with data on what it spends on external probity practitioners. Transport for NSW spent over one third of the $16.2 million total estimated amount spent by 37 agencies over the last three financial years. Some of these agencies acknowledged their data may not be complete and accurate.

To promote transparency and accountability across the public sector, those charged with the governance of agencies should ask agency management to periodically report on the use and cost of engaging probity practitioners. Agencies should ensure they have the necessary systems in place to collect and report this information accurately and completely.

2. Effective use of probity advisers and probity auditors

2.1 Value for money assessments

Section 176 of the Public Works and Procurement Act 1912 (the Act), imposes a statutory obligation on agencies to ensure they obtain value for money when they procure goods and services.

Similarly, the NSW Procurement Board is obliged, under section 171 of the Act, to ensure it obtains best value for money when procuring goods and services by and for government agencies. 

The Board's 'Value for Money Statement' defines value for money as 'the difference between the total benefit derived from a good or a service against its total cost, when assessed over the period the goods or services are to be used'. The Statement also includes detailed guidelines to help agencies assess value for money. 

Agencies could not demonstrate engaging probity practitioners achieved quality outcomes

New South Wales has no regulated probity standards, principles or frameworks that apply to the work of probity practitioners. As a result, the nature and quality of the procedures different practitioners perform varies. This in turn diminishes the degree of reliance stakeholders can place on practitioners’ work and may result in poor decision making.

In previous performance audit reports, we have identified probity issues associated with various transactions entered into by government agencies.These include:

  • ineffective management of conflicts of interests
  • independence issues with the probity advisers
  • expectation gaps on probity scopes of work
  • poor record keeping.

In this audit, we found instances in all three participating agencies where they had:

  • not agreed terms of engagement that detailed the scope of the work and project specific risks. In some cases, the agency relied on the probity practitioner to document the terms of reference for them
  • not undertaken effective value for money assessments. Some agencies asserted they had achieved the desired outcomes and value for money on low value probity engagements if the probity practitioner provided a clear report. The top 40 surveyed agencies gave varying responses as to how they perceived value for money
  • not formally evaluated or assessed whether the probity engagements had achieved the desired outcomes
  • not evaluated probity practitioners' performance against engagement scopes to determine if the services rendered represented value for money. Some agencies do not believe they should do so for low value probity engagements because the NSW Procurement Board does not require performance assessments for suppliers, selected from the PMS scheme, costing less than $150,000. In our view, post engagement assessments give agencies opportunities to gather learnings that may help them build internal probity expertise. 

Exhibit 5 below compares those considerations the 40 surveyed agencies apply when assessing value for money with considerations we believe were not sufficiently covered in agencies assessment processes.

The following diagram shows the areas currently considered and insufficiently considered by the top 40 agencies when assessing value for money. For an accessible version of this diagram email communications@audit.nsw.gov.au
Exhibit 5: Compares considerations the top 40 agencies apply when assessing value for money with considerations they should apply to enhance their assessments

2.2 Achieving quality probity outcomes through timely engagement

Agencies engage probity practitioners at different stages of their projects

Engaging probity practitioners in a timely manner can help ensure agencies identify and consider probity matters, risks and issues early in a project's life. We found instances where the participating agencies engaged probity practitioners at different phases of the procurement process. 

In one instance, the Ministry of Health only engaged a probity practitioner after its procurement strategy, to enter into a direct negotiation with a single supplier, had been approved. The Ministry advises it had no other options as the only other two qualified suppliers in the market were already engaged on projects. However, the Ministry did not ask the probity practitioner to review the Ministry's rationale for entering into a direct negotiation procurement. The probity adviser, who was engaged after the direct negotiation commenced, recommended in the probity report that they should have been engaged earlier to ensure potential probity issues at the procurement strategy stage were properly addressed.

In another instance, the Department of Education only engaged a probity adviser after it had sent requests for quotation to selected suppliers. It did not have a probity plan to support the procurement process. 

2.3 Importance of record keeping and access to key information

Agencies did not give probity practitioners sufficient access to records and meetings

Agencies should give probity practitioners full authority to access records, personnel, meetings and premises so they have all the information they need to fully discharge their duties and exercise due diligence in a procurement or sale process.

We found an instance where the Department of Education's probity practitioner for a major school infrastructure project:

  • was unable to access all of the Department's records
  • was not present at all key meetings for all stages of the procurement
  • did not obtain and review key conflict of interest declarations from procurement tender participants.

In another instance, the Ministry of Health engaged a probity adviser after key meetings had occurred and did not maintain sufficient records for the probity adviser to do a post engagement review of the meeting minutes.

Probity practitioners did not have independent reporting lines

Probity practitioners need independent reporting lines so they can bypass management and report to those charged with governance, internal audit and external watchdog organisations, if necessary. Independent reporting lines help ensure the independence and authority of probity practitioners. An administrative reporting line to project managers should be used to facilitate day to day probity activities and provide an appropriate interface and effective support. Exhibit 6 shows the ideal administrative and independent reporting lines for probity practitioners. 

This flow diagram shows that there is a direct administrative reporting line between project manager and probity practitioner and an independent reporting line between probity practitioner and the internal auditor and people charged with governance. For accessible version email communications@audit.nsw.gov.au
Exhibit 6: A typical probity practitioner's administrative and independent reporting lines

We found instances at all three participating agencies where:

  • probity practitioners' reporting lines had not been defined in the terms of engagement
  • probity practitioners reported directly to the project manager
  • probity practitioners did not have independent reporting lines to those charged with governance and internal audit.

Agencies’ audit and risk committees play an important role in probity governance and related processes. These committees can ensure internal audit programs periodically review that agencies' procurement and related probity processes meet compliance requirements, effectively manage and mitigate risks and achieve value for money.

Agencies should give probity practitioners independent reporting lines so they can effectively discharge their responsibilities. 

2.4 Better government support for agencies

Agencies need better probity guidance and support from central agencies

Agencies would benefit from probity guidance to help them consistently and effectively manage and mitigate probity risks associated with complex and unique asset procurement and disposal transactions.

The gaps in probity standards, regulations or best practice guides in probity mean agencies do not have a clear and comprehensive probity framework to help them apply probity principles throughout project decision making processes and meet stakeholder expectations.

The probity considerations in projects plays an important role in the New South Wales gateway review process. The NSW Government has a gateway review process for capital, recurrent and information and communication technology projects. The process involves independent expert reviews of up to seven defined decision points (or gates) in a project or program's life cycle. The reviews assess the robustness of project proposals and contribute to improving ultimate performance. For example, the gate four review in the 'NSW Infrastructure Investor Assurance Framework' requires the delivery agency to have probity plans in place and demonstrate compliance with the plans.

A coordinated effort by the Department of Premier and Cabinet and Gateway Coordination Agencies is needed to support agencies manage probity engagements, achieve better probity outcomes and ensure engagements represent value for money. Gateway Coordination Agencies are responsible for designing and administering a Gateway Coordination Framework for capital, recurrent and information and communication technology projects. This gives other agencies opportunities to leverage their expertise to improve probity management.

The NSW Procurement Board also has a role in setting minimum expectations through the pre qualification scheme for probity providers' qualifications, capabilities and experience in probity risk management.