Reports
Ensuring teaching quality in NSW public schools
The Auditor-General for New South Wales, Margaret Crawford, has released a report on how the New South Wales Education and Standards Authority (NESA) and the Department of Education (the Department) ensure teaching quality in NSW public schools.
Around 2,200 NSW public school principals are responsible for accrediting their teachers in line with the Australian Professional Standards for Teachers. The report found that NESA does not oversight principals’ decisions to ensure that minimum standards for teaching quality are consistently met.
The Department does not effectively monitor teaching quality across the state. With limited data, it is difficult for the Department to ensure its strategies to improve teaching quality are appropriately targeted to improve teaching quality.
The Department’s Performance and Development Framework does not adequately support principals and supervisors to effectively manage and improve teacher performance or actively improve teaching quality. The Department manages those teachers formally identified as underperforming through teacher improvement programs. Only 53 of over 66,000 teachers employed by the Department were involved in these programs in 2018.
The report makes three recommendations towards NESA to improve accreditation processes, and four recommendations to the Department to improve its systems and processes for ensuring teaching quality across the State.
Australian research has shown that quality teaching is the greatest in-school influence on student engagement and outcomes, accounting for 30 per cent of the variance in student performance. An international comparative study of 15-year-old students showed the performance of New South Wales students in reading, mathematics and science has declined between 2006 and 2015.
The Australian Professional Standards for Teachers (the Standards) describe the knowledge, skills and understanding expected of effective teachers at different career stages. Teachers must be accredited against the Standards to be employed in NSW schools. The NSW Education Standards Authority (NESA) is responsible for ensuring all teachers in NSW schools are accredited. As part of the accreditation process the NSW Department of Education (The Department) assesses whether public school teachers meet proficient accreditation standards and advises NESA of its decisions.
The School Excellence Framework provides a method for the Department to monitor teaching quality at a school level across four elements of effective teaching practice. The Performance and Development Framework provides a method for teachers and their supervisors to monitor and improve teaching quality through setting professional goals to guide their performance and development.
The Department has a strategic goal that every student, every teacher, every leader and every school improves every year. In line with this goal, the Department has a range of strategies targeted to improving teaching quality at different career stages. These include additional resources to support new teachers, a program to support teachers to gain higher-level accreditation, support for principals to manage underperforming teachers, and a professional learning program where teachers observe and discuss each other's practice.
The objective of this audit was to assess the effectiveness of the NSW Department of Education's and the NSW Education Standards Authority's arrangements to ensure teaching quality in NSW public schools. To address this objective, the audit examined whether:
- agencies effectively monitor the quality of teaching in NSW public schools
- strategies to improve the quality of teaching are planned, communicated, implemented and monitored well.
Appendix one – Response from agencies
Appendix two – About the audit
Appendix three – Performance auditing
© Copyright reserved by the Audit Office of New South Wales. All rights reserved. No part of this publication may be reproduced without prior consent of the Audit Office of New South Wales. The Audit Office does not accept responsibility for loss or damage suffered by any person acting on or refraining from action as a result of any of this material.
Parliamentary Reference: Report number #327 - released 26 September 2019
Mental health service planning for Aboriginal people in New South Wales
A report released by the Auditor-General for New South Wales, Margaret Crawford, has found that NSW Health is not forming effective partnerships with Aboriginal communities to plan, design and deliver appropriate mental health services. There is limited evidence that NSW Health is using the knowledge and expertise of Aboriginal communities to guide how mental health care is structured and delivered.
Mental illness (including substance use disorders) is the main contributor to lower life expectancy and increased mortality in the Aboriginal population of New South Wales. It contributes to a higher burden of disease and premature death at rates that are 40 per cent higher than the next highest chronic disease group, cardiovascular disease.1
Aboriginal people have significantly higher rates of mental illness than non Aboriginal people in New South Wales. They are more likely to present at emergency departments in crisis or acute phases of mental illness than the rest of the population and are more likely to be admitted to hospital for mental health treatments.2
In acknowledgement of the significant health disparities between Aboriginal and non Aboriginal people, NSW Health implemented the NSW Aboriginal Health Plan 2013 2023 (the Aboriginal Health Plan). The overarching message of the Aboriginal Health Plan is ‘to build respectful, trusting and effective partnerships with Aboriginal communities’ and to implement ‘integrated planning and service delivery’ with sector partners. Through the Plan, NSW Health commits to providing culturally appropriate and ‘holistic approaches to the health of Aboriginal people'.
The mental health sector is complex, involving Commonwealth, state and non government service providers. In broad terms, NSW Health has responsibility to support patients requiring higher levels of clinical support for mental illnesses, while the Commonwealth and non government organisations offer non acute care such as assessments, referrals and early intervention treatments.
The NSW Health network includes 15 Local Health Districts and the Justice Health and Forensic Mental Health Network that provide care to patients during acute and severe phases of mental illness in hospitals, prisons and community service environments. This includes care to Aboriginal patients in the community at rates that are more than four times higher than the non Aboriginal population. Community services are usually provided as follow up after acute admissions or interactions with hospital services. The environments where NSW Health delivers mental health care include:
- hospital emergency departments, for short term assessment and referral
- inpatient hospital care for patients in acute and sub acute phases of mental illness
- mental health outpatient services in the community, such as support with medications
- custodial mental health services in adult prisons and juvenile justice centres.
The NSW Government is reforming its mental health funding model to incrementally shift the balance from hospital care to enhanced community care. In 2018–19, the NSW Government committed $400 million over four years into early intervention and specialist community mental health teams.
This audit assessed the effectiveness of NSW Health’s planning and coordination of mental health services and service pathways for Aboriginal people in New South Wales. We addressed the audit objective by answering three questions:
- Is NSW Health using evidence to plan and inform the availability of mental health services for Aboriginal people in New South Wales?
- Is NSW Health collaborating with partners to create accessible mental health service pathways for Aboriginal people?
- Is NSW Health collaborating with partners to ensure the appropriateness and quality of mental health services for Aboriginal people?
NSW Health is not meeting the objectives of the NSW Aboriginal Health Plan, to form effective partnerships with Aboriginal Community Controlled Health Services and Aboriginal communities to plan, design and deliver mental health services.
There is limited evidence that existing partnerships between NSW Health and Aboriginal communities meet its own commitment to use the ‘knowledge and expertise of the Aboriginal community (to) guide the health system at every level, including (for) the identification of key issues, the development of policy solutions, the structuring and delivery of services' 3 and the development of culturally appropriate models of mental health care.
NSW Health is planning and coordinating its resources to support Aboriginal people in acute phases of mental illness in hospital environments. However, it is not effectively planning for the supply and delivery of sufficient mental health services to assist Aboriginal patients to manage mental illness in community environments. Existing planning approaches, data and systems are insufficient to guide the $400 million investment into community mental health services announced in the 2018–19 Budget.
NSW Health is not consistently forming partnerships to ensure coordinated care for patients as they move between mental health services. There is no policy to guide this process and practices are not systematised or widespread.
In this report, the term ‘Aboriginal people’ is used to describe both Aboriginal and Torres Strait Islander peoples. The Audit Office of NSW acknowledges the diversity of traditional countries and Aboriginal language groups across the state of New South Wales.
2 Australian Institute of Health and Welfare data 2016–17 (unaudited).
3 NSW Health, The Aboriginal Health Plan 2013-2023.
In May 2019, the Audit Office of New South Wales invited Aboriginal mental health clinicians and policy experts from government and non-government organisations to attend a one-day workshop. Workshop attendees advised on factors that improve the quality and appropriateness of mental health care for Aboriginal people in New South Wales. They described appropriate mental health care as:
- culturally safe, allowing Aboriginal people to draw strength in their identity, culture and community
- person centred and focussed on individual needs
- delivered by culturally competent staff with no bias
- holistic, trauma-informed and focussed on early intervention where possible
- delivered in places that are appropriate including outreach to homes and communities
- welcoming of the involvement of local Aboriginal community and connected to local knowledge and expertise including totems and kinship structures.
The definition of 'appropriate' mental health care for Aboriginal people throughout this report is based on this advice.
Aboriginal people access emergency services at much higher rates than non-Aboriginal people
The choices that people make in relation to health service options provide some insight into the suitability and appropriateness of the service to their needs.
Aboriginal people have different mental health service use patterns than non-Aboriginal people. Aboriginal people are much more likely to be in a crisis situation before receiving mental health services, usually in an emergency department of a hospital.
Aboriginal people make up three per cent of the total New South Wales population, but they constitute 11 per cent of emergency department presentations for mental health treatments. In regional areas, Aboriginal people make up 20.5 per cent of presentations at emergency departments for mental health reasons.
A number of factors help to explain Aboriginal mental health service usage patterns. According to government and non-government mental health organisations:
- emergency department services are better known to Aboriginal people than other mental health services
- community-based models of care are not appropriate for Aboriginal people
- Aboriginal people are reluctant to access community-based mental health services to prevent crisis situations
- community mental health services are not available for Aboriginal people after hours and during the weekend, so emergency services are the only option.
The statewide proportions of Aboriginal people presenting at emergency departments for mental health treatments has been increasing over time (Exhibit 6).
Appendix one – Response from agency
Appendix two – The NSW Aboriginal Health Plan
Appendix three – About the audit
Appendix four – Performance auditing
Parliamentary Reference: Report number #326 - released 29 August 2019
Copyright reserved by the Audit Office of New South Wales. All rights reserved. No part of this publication may be reproduced without prior consent of the Audit Office of New South Wales. The Audit Office does not accept responsibility for loss or damage suffered by any person acting on or refraining from action as a result of any of this material.
Ensuring contract management capability in government - Department of Education
This report examines whether the Department of Education has the required contract management capability to effectively manage high-value goods and services contracts (over $250,000). In 2017–18, the department managed high-value goods and services contracts worth $3.08 billion, with most of the contracts running over multiple years.
NSW government agencies are increasingly delivering services and projects through contracts with third parties. These contracts can be complex and governments face challenges in negotiating and implementing them effectively.
Contract management capability is a broad term, which can include aspects of individual staff capability as well as organisational capability (such as policies, frameworks and processes).
In 2017–18, the Department of Education (the Department) managed high-value (over $250,000) goods and services contracts worth $3.08 billion, with most of the contracts running over multiple years. The Department delivers, funds and regulates education services for NSW students from early childhood to secondary school.
This audit examined whether the Department has the required capability to effectively manage high-value goods and services contracts.
We did not examine infrastructure, construction or information communication and technology contracts. We assessed the Department against the following criteria:
- The Department’s policies and procedures support effective contract management and are consistent with relevant frameworks, policies and guidelines.
- The Department has capable personnel to effectively conduct the monitoring activities throughout the life of the contract.
The NSW Public Service Commission and the Department of Finance, Services and Innovation are included as auditees as they administer policies which directly affect contract management capability, including:
- NSW Procurement Board Directions and policies
- NSW Procurement Agency Accreditation Scheme
- NSW Public Sector Capability Framework.
The Department of Finance, Services and Innovation's responsibility for NSW Procurement will transfer to NSW Treasury on 1 July 2019 as part of changes to government administrative arrangements announced on 2 April 2019 and amended on 1 May 2019.
|
Conclusion The Department of Education's procedures and policies for goods and services contract management are consistent with relevant guidance. It also has a systemic approach to defining the capability required for contract management roles. That said, there are gaps in how well the Department uses this capability to ensure its contracts are performing. We also found one program (comprising 645 contracts) that was not compliant with the Department's policies. The Department has up-to-date policies and procedures that are consistent with relevant guidance. The Department also communicates changes to procurement related policies, monitors compliance with policies and conducts regular reviews aiming to identify non-compliance. The Department uses the NSW Public Service Commission's capability framework to support its workforce management and development. The capability framework includes general contract management capability for all staff and occupation specific capabilities for contract managers. The Department also provides learning and development for staff who manage contracts to improve their capability. The Department provides some guidance on different ways that contract managers can validate performance information provided by suppliers. However, the Department does not provide guidance to assist contract managers to choose the best validation strategy according to contract risk. This could lead to inconsistent practice and contracts not delivering what they are supposed to. We found that none of the 645 contracts associated with the Assisted Schools Travel Program (estimated value of $182 million in 2018–19) have contract management plans. This is contrary to the Department's policies and increases the risk that contract managers are not effectively reviewing performance and resolving disputes. |
Appendix one - Response from agencies
Appendix two - About the audit
Appendix three - Performance auditing
Parliamentary Reference: Report number #325 - released 28 June 2019
Copyright reserved by the Audit Office of New South Wales. All rights reserved. No part of this publication may be reproduced without prior consent of the Audit Office of New South Wales. The Audit Office does not accept responsibility for loss or damage suffered by any person acting on or refraining from action as a result of any of this material.
Wellbeing of secondary school students
The Department of Education has a strong focus on supporting secondary school students’ wellbeing. However, it is difficult to assess how well the Department is progressing as it is yet to measure or report on the outcomes of this work at a whole-of-state level.
The Department of Education’s (the Department) purpose is to prepare young people for rewarding lives as engaged citizens in a complex and dynamic society. The Department commits to creating quality learning opportunities for children and young people, including a commitment to student wellbeing, which is seen as directly linked to positive learning outcomes. Wellbeing is defined broadly by the Department as “the quality of a person’s life…It is more than the absence of physical or psychological illness”. Student wellbeing can be supported by everything a school does to enhance a student's learning—from curriculum to teacher quality to targeted policies and programs to whole-school approaches to wellbeing.
Several reforms have aimed to support student wellbeing in recent years. 'Local Schools, Local Decisions' gave NSW schools more local authority to make decisions, including schools' approaches to support student wellbeing. In 2016, the 'Supported Students, Successful Students' initiative provided $167 million over four years to support the wellbeing of students. From 2018, the 'Every Student is Known, Valued and Cared For' initiative provides a principal led mentoring program, and a website with policies, procedures and resources to support student wellbeing.
This audit assessed how well the Department of Education supports secondary schools to promote and support the wellbeing of their students and how well secondary schools are promoting and supporting the wellbeing of their students.
Conclusion
The Department has implemented a range of programs and reforms aimed at supporting student wellbeing. However, the outcomes of this work have yet to be measured or reported on at a system level, making it difficult to assess the Department's progress in improving student wellbeing.
Secondary schools have generally adopted a structured approach to deliver wellbeing support and programs, using both Department and localised resources. The approaches have been tailored to meet the needs of their school community. That said, public reporting on wellbeing improvement measures via annual school reports is of variable quality and needs to improve.
The Department’s wellbeing initiatives are supported by research and consultation, but outcomes have not been reported on
The Department’s development of wellbeing policy, guidance, tools and resources has been transparent, consultative and well researched. It has drawn on international and domestic evidence to support its aim to deliver a fundamental shift from welfare to wellbeing at the school and system level.
However, the key performance indicator to monitor and track progress in wellbeing has yet to be reported on despite the strategic plan including this as a priority for the period 2018 to 2022. This includes not yet reporting a baseline for the target, nor how it will be measured.
The Department’s wellbeing resources are mostly well targeted but there is room for improvement
The Department’s allocation of resources to deliver wellbeing initiatives in schools is mostly well targeted, reflects a needs basis and supports current strategic directions. This could be improved with some changes to formula allocations and clearer definitions of the resourcing required for identified wellbeing positions in schools. The workforce modelling for forecasting supply and demand, specifically for school counsellors and psychologists, needs to separately identify these positions as they are currently subsumed in general teacher numbers.
Schools' reporting on wellbeing improvement measures is of variable quality and needs to improve
Schools we visited demonstrated a variety of approaches to wellbeing depending on their local circumstances and student populations. They make use of Department policies, guidelines, and resources, particularly mandatory policies and data collections, which have good compliance and take-up at school level. Professional learning supports specific wellbeing initiatives and online systems for monitoring and reporting have contributed to schools’ capacity and capabilities.
Schools report publicly on wellbeing improvement measures through annual school reports but this reporting is of variable quality. The Department plans to improve the capability of schools in data analysis and we recommend that this include the setting and evaluation of improvement targets for wellbeing.
The implementation of the 2015 Wellbeing Framework in schools is incomplete and the Department has not effectively prioritised and consolidated tools, systems and reporting for wellbeing
Schools' take up of the 2015 Wellbeing Framework is hindered by it not being linked to the school planning and reporting policy and tools—the School Excellence Framework. At some schools we visited, this disconnect has led to a lack of knowledge and confidence in using it in schools. The Department has identified the need to improve alignment of policies, frameworks and plans and has commenced work on this.
We found evidence of overburdening in schools for addressing student wellbeing—in the number of tools, online systems for information collection, and duplication in reporting. Following the significant reforms of recent years, the Department should consolidate its efforts by reinforcing existing effective programs and systems and addressing identified gaps and equity issues, rather than introducing further change for schools. In particular, methods and processes for complex case coordination need improvement.
The NSW Department of Education commits to creating quality learning opportunities for students. This includes strengthening students’ physical, social, emotional and spiritual development. The Department sets out to enable students to be healthy, happy, engaged and successful.
Welfare and wellbeing
The Department’s approach has significantly shifted from student welfare to wellbeing of the whole child and young person. Wellbeing is defined in departmental policy and strategy documents broadly, and as directly linked to learning and positive learning outcomes. “Wellbeing can be described as the quality of a person’s life…It is more than the absence of physical or psychological illness…Wellbeing, or the lack of it, can affect a student’s engagement and success in learning…”
Student wellbeing can be supported by everything a school does to enhance a student's learning—from curriculum to teacher quality to targeted policies and programs to whole-school approaches to wellbeing. Distinctions between wellbeing and welfare in the school context are outlined below.
| Welfare | Wellbeing |
|---|---|
| Operates from a basis of student need and doesn't always take into account a whole child view. | For all students. |
| Rather than building on the strengths of students, operates from a deficit model of individual student problems or negative behaviours. | Goes beyond just welfare needs of a few students and aims for all students to be healthy, happy, successful and productive individuals who are active and positive contributors to the school and society in which they live. |
Appendix one - Response from agency
Appendix two - Key policies, guidance, and systems
Appendix three - Funding and resources for schools
Appendix four - Measuring wellbeing
Appendix five - About the audit
Appendix six - Performance auditing
Parliamentary Reference: Report number #318 - released 23 May 2019
Supply of secondary teachers in STEM-related disciplines
The NSW Department of Education’s plans and strategies to respond to the demand for secondary teachers in STEM-related disciplines are limited by incomplete data and underperforming scholarship and sponsorship program. The Department does not collect sufficient information to monitor what disciplines teachers actually teach nor does it predict supply and demand for teachers by discipline and location. This restricts the Department’s ability to track and forecast the supply and demand for secondary teachers in STEM-related disciplines.
In recent years, Australian and international education policy has focused on improving outcomes in Science, Technology, Engineering and Mathematics (STEM) subjects. However, research has identified a shortage of qualified secondary teachers in STEM-related disciplines 1. This is projected to worsen due to a combination of student population increases, an ageing workforce, and fewer people going into teaching. Shortfalls are likely to be more acute in rural and remote areas, and areas of low socio-economic status.
The Department of Education (the Department) has a variety of strategies to encourage teachers to practise in locations or disciplines of need. These include scholarships for tertiary students going into teaching, sponsorships for teachers seeking approval to teach additional disciplines, and incentives to attract teachers to rural and remote locations.
This audit assessed the effectiveness of the Department's workforce plans and strategies in responding to the demand for secondary teachers in STEM-related disciplines. We assessed:
- how well the Department tracks the supply and demand for secondary teachers in STEM-related disciplines across NSW
- whether the Department has effective strategies to attract and retain secondary teachers in STEM-related disciplines.
There are two key shortcomings that fundamentally limit the effectiveness of the Department's plans and strategies to respond to the demand for secondary teachers in STEM-related disciplines. First, the Department is not accurately tracking the supply and demand for secondary teachers by discipline due to incomplete data. Second, not all scholarship and sponsorship places are allocated and many scholars withdraw from the programs before completion. The Department has recognised and started to address these problems with a new workforce model, revised incentives and scholarship programs.
The Department’s current workforce planning model does not provide the information needed to target workforce plans and strategies to areas of need. This is because it does not predict supply and demand for teachers by discipline and location. An internal review in 2017 acknowledged the limitations of this model. In response the Department developed a new model, which it is currently enhancing, to predict supply and demand for teachers by discipline and location. For this to be successful, the Department needs to monitor the level of out-of-field teaching and improve data on the willingness of teachers to work in particular locations.
The Department does not allocate all available scholarship and sponsorship places and around 30 per cent of recipients do not complete the term of their agreement. An internal review in 2017 highlighted that some programs were not targeting workforce need and that there were no key performance indicators to determine the overall effectiveness of these programs. However, scholarship programs and incentives are promoted well through social media and face-to-face events at Universities. Further, the Department has used findings from internal reviews of incentives and scholarships in 2016 and 2017 to inform recent changes to programs.
The Department has little oversight of access to practicum placements for pre-service teachers in areas of need. Professional experience agreements were established with each University in 2015 to improve the placement process for disciplines of need. Initial teacher education students must complete several ‘practicum placements’ before they can be qualified to teach in a school. Several universities we consulted reported difficulties finding practicum placements for pre-service teachers specialising in STEM-related disciplines. The Department is now revising the agreements to improve the quality of data it collects on the number, location and subject area of practicum placements.
The Department is not accurately tracking the supply and demand for secondary teachers by discipline due to incomplete data.
The Department’s current workforce planning model does not accurately predict supply and demand for teachers by discipline and location. An internal review in 2017 acknowledged the limitations of this model. In response the Department developed a new model which it is currently enhancing to address the findings of the review. For this model to be successful, the Department needs to monitor the level of out-of-field teaching and improve data on the willingness of teachers to work in particular locations. Further work also needs to be undertaken to refine the assumptions that underpin the Department’s workforce planning models as it starts to predict the need for teachers by discipline.
The Department has not publicly reported on the supply and demand for teachers by discipline since 2015. While it does report annually on its current workforce profile, this information is not detailed enough to inform future strategies or programs. More detailed public reporting may help the Department to influence the future supply of teachers by communicating its projected areas of need. Planned improvements to the Department's workforce planning model, as relayed to us, will add to the data available on areas of need. Once available, this should be reported publicly.
Recommendations
By December 2019, the Department of Education should:
- Improve its workforce planning model to better understand and communicate supply and demand for teachers by:
- determining the extent, and analysing the impact, of out-of-field teaching by permanent and temporary teachers in each school
- sourcing additional data to more accurately reflect teacher location preferences
- projecting supply and demand by subject level and geographic area
- regularly reporting on the supply and demand for secondary teachers in each discipline to communicate future areas of need to future teacher education students.
The Department's current scholarship and sponsorship programs are not allocating all available places and many scholars withdraw from the programs before completion. An internal review in 2017 raised several issues with the effectiveness of programs and the Department has started to revise its scholarship, sponsorship and incentive programs.
An internal review in 2017 highlighted that scholarship and sponsorship programs were not targeting workforce need, and that there were no key performance indicators to determine the overall effectiveness of these strategies. In addition, the review found that only 79 per cent of available scholarship placements are allocated each year, and 31 per cent of scholarship recipients withdraw prior to completing their required service period. The Department recently announced changes to its scholarship programs from 2019 onwards.
The Department has incentives to encourage teachers to work in rural and remote areas, including teachers in STEM-related disciplines. Incentives include access to priority transfers, rental subsidies and other allowances. Research conducted in 2016 examined the influence of incentives in encouraging teachers to work in rural and remote areas. The Department used findings of this research when updating its set of rural and remote incentives in 2017.
The Department promotes its scholarship and sponsorship programs through the teach.NSW website. It uses social media to direct applicants to this website. It also promotes its programs through careers fairs, University open days, and professional events. Past applicants have reported that the website clearly communicates eligibility criteria and the terms of agreement for all scholarship programs.
The Department could strengthen its relationship with universities to attract teachers to areas of need by collecting and analysing data on practicum placements, facilitating placements for scholarship recipients, and communicating predicted teacher needs by discipline.
Recommendations
By December 2019, the Department of Education should:
2. Implement changes to address the findings of the 'Teacher Scholarship Realignment' report, including by:
- testing a range of program designs with target candidates to determine the best options to attract more suitable applicants
- establishing key performance indicators, and setting targets, to better monitor the effectiveness of the programs
- reducing the number of scholars appointed to over-establishment positions
- increasing the proportion of scholars appointed to priority locations
- further analysing scholarship recipients career paths to inform future improvements to the scholarship programs.
3. Review its role in the practicum placement process of pre-service teachers by:
- analysing how many students each school accommodates per year, to ensure there are appropriate placements available for students in high needs disciplines
- working with universities to facilitate practicum placements for scholarship recipients
- establishing mechanisms for ongoing monitoring of its partnerships with universities to ensure they are meeting their aims.
Appendix one - Response from agency
Appendix two - About the audit
Appendix three - Performance auditing
Parliamentary Reference - Report number #313 - released 29 January 2019.
HealthRoster benefits realisation
The HealthRoster system is delivering some business benefits but Local Health Districts are yet to use all of its features, according to a report released today by the Auditor-General for New South Wales, Margaret Crawford. HealthRoster is an IT system designed to more effectively roster staff to meet the needs of Local Health Districts and other NSW health agencies.
The NSW public health system employs over 100,000 people in clinical and non-clinical roles across the state. With increasing demand for services, it is vital that NSW Health effectively rosters staff to ensure high quality and efficient patient care, while maintaining good workplace practices to support staff in demanding roles.
NSW Health is implementing HealthRoster as its single state-wide rostering system to more effectively roster staff according to the demands of each location. Between 2013–14 and 2016–17, our financial audits of individual LHDs had reported issues with rostering and payroll processes and systems.
NSW Health grouped all Local Health Districts (LHDs), and other NSW Health organisations, into four clusters to manage the implementation of HealthRoster over four years. Refer to Exhibit 4 for a list of the NSW Health entities in each cluster.
- Cluster 1 implementation commenced in 2014–15 and was completed in 2015–16.
- Cluster 2 implementation commenced in 2015–16 and was completed in 2016–17.
- Cluster 3 began implementation in 2016–17 and was underway during the conduct of the audit.
- Cluster 4 began planning for implementation in 2017–18.
Full implementation, including capability for centralised data and reporting, is planned for completion in 2019.
This audit assessed the effectiveness of the HealthRoster system in delivering business benefits. In making this assessment, we examined whether:
- expected business benefits of HealthRoster were well-defined
- HealthRoster is achieving business benefits where implemented.
The HealthRoster project has a timespan from 2009 to 2019. We examined the HealthRoster implementation in LHDs, and other NSW Health organisations, focusing on the period from 2014, when eHealth assumed responsibility for project implementation, to early 2018.
Business benefits identified for HealthRoster accurately reflect business needs.
NSW Health has a good understanding of the issues in previous rostering systems and has designed HealthRoster to adequately address these issues. Interviews with frontline staff indicate that HealthRoster facilitates rostering which complies with industrial awards. This is a key business benefit that supports the provision of quality patient care. We saw no evidence that any major business needs or issues with the previous rostering systems are not being addressed by HealthRoster.
In the period examined in this audit since 2015, NSW Health has applied appropriate project management and governance structures to ensure that risks and issues are well managed during HealthRoster implementation.
HealthRoster has had two changes to its budget and timeline. Overall, the capital cost for the project has increased from $88.6 million to $125.6 million (42 per cent) and has delayed expected project completion by four years from 2015 to 2019. NSW Health attributes the increased cost and extended time frame to the large scale and complexity of the full implementation of HealthRoster.
NSW Health has established appropriate governance arrangements to ensure that HealthRoster is successfully implemented and that it will achieve business benefits in the long term. During implementation, local steering committees monitor risks and resolve implementation issues. Risks or issues that cannot be resolved locally are escalated to the state-wide steering committee.
NSW Health has grouped local health districts, and other NSW Health organisations, into four clusters for implementation. This has enabled NSW Health to apply lessons learnt from each implementation to improve future implementations.
NSW Health has a benefits realisation framework, but it is not fully applied to HealthRoster.
NSW Health can demonstrate that HealthRoster has delivered some functional business benefits, including rosters that comply with a wide variety of employment awards.
NSW Health is not yet measuring and tracking the value of business benefits achieved. NSW Health did not have benefits realisation plans with baseline measures defined for LHDs in cluster 1 and 2 before implementation. Without baseline measures NSW Health is unable to quantify business benefits achieved. However, analysis of post-implementation reviews and interviews with frontline staff indicate that benefits are being achieved. As a result, NSW Health now includes defining baseline measures and setting targets as part of LHD implementation planning. It has created a benefits realisation toolkit to assist this process from cluster 3 implementations onwards.
NSW Health conducted post-implementation reviews for clusters 1 and 2 and found that LHDs in these clusters were not using HealthRoster to realise all the benefits that HealthRoster could deliver.
By September 2018, NSW Health should:
- Ensure that Local Health Districts undertake benefits realisation planning according to the NSW Health benefits realisation framework
- Regularly measure benefits realised, at state and local health district levels, from the statewide implementation of HealthRoster
- Review the use of HealthRoster in Local Health Districts in clusters 1 and 2 and assist them to improve their HealthRoster related processes and practices.
By June 2019, NSW Health should:
- Ensure that all Local Health Districts are effectively using demand based rostering.
Appendix one - Response from agency
Appendix two - About the audit
Appendix three - Performance auditing
Parliamentary reference - Report number #301 - released 7 June 2018
Managing risks in the NSW public sector: risk culture and capability
The Ministry of Health, NSW Fair Trading, NSW Police Force, and NSW Treasury Corporation are taking steps to strengthen their risk culture, according to a report released today by the Auditor-General, Margaret Crawford. 'Senior management communicates the importance of managing risk to their staff, and there are many examples of risk management being integrated into daily activities', the Auditor-General said.
We did find that three of the agencies we examined could strengthen their culture so that all employees feel comfortable speaking openly about risks. To support innovation, senior management could also do better at communicating to their staff the levels of risk they are willing to accept.
Effective risk management is essential to good governance, and supports staff at all levels to make informed judgements and decisions. At a time when government is encouraging innovation and exploring new service delivery models, effective risk management is about seizing opportunities as well as managing threats.
Over the past decade, governments and regulators around the world have increasingly turned their attention to risk culture. It is now widely accepted that organisational culture is a key element of risk management because it influences how people recognise and engage with risk. Neglecting this ‘soft’ side of risk management can prevent institutions from managing risks that threaten their success and lead to missed opportunities for change, improvement or innovation.
This audit assessed how effectively NSW Government agencies are building risk management capabilities and embedding a sound risk culture throughout their organisations. To do this we examined whether:
- agencies can demonstrate that senior management is committed to risk management
- information about risk is communicated effectively throughout agencies
- agencies are building risk management capabilities.
The audit examined four agencies: the Ministry of Health, the NSW Fair Trading function within the Department of Finance, Services and Innovation, NSW Police Force and NSW Treasury Corporation (TCorp). NSW Treasury was also included as the agency responsible for the NSW Government's risk management framework.
In assessing an agency’s risk culture, we focused on four key areas:
Executive sponsorship (tone at the top)
In the four agencies we reviewed, senior management is communicating the importance of managing risk. They have endorsed risk management frameworks and funded central functions tasked with overseeing risk management within their agencies.
That said, we found that three case study agencies do not measure their existing risk culture. Without clear measures of how employees identify and engage with risk, it is difficult for agencies to tell whether employee's behaviours are aligned with the 'tone' set by the executive and management.
For example, in some agencies we examined we found a disconnect between risk tolerances espoused by senior management and how these concepts were understood by staff.
Employee perceptions of risk management
Our survey of staff indicated that while senior leaders have communicated the importance of managing risk, more could be done to strengthen a culture of open communication so that all employees feel comfortable speaking openly about risks. We found that senior management could better communicate to their staff the levels of risk they should be willing to accept.
Integration of risk management into daily activities and links to decision-making
We found examples of risk management being integrated into daily activities. On the other hand, we also identified areas where risk management deviated from good practice. For example, we found that corporate risk registers are not consistently used as a tool to support decision-making.
Support and guidance to help staff manage risks
Most case study agencies are monitoring risk-related skills and knowledge of their workforce, but only one agency has addressed the gaps it identified. While agencies are providing risk management training, surveyed staff in three case study agencies reported that risk management training is not adequate.
NSW Treasury provides agencies with direction and guidance on risk management through policy and guidelines. In line with better practice, NSW Treasury's principles-based policy acknowledges that individual agencies are in a better position to understand their own risks and design risk management frameworks that address those risks. Nevertheless, there is scope for NSW Treasury to refine its guidance material to support a better risk culture in the NSW public sector.
Recommendation
By May 2019, NSW Treasury should:
- Review the scope of its risk management guidance, and identify additional guidance, training or activities to improve risk culture across the NSW public sector. This should focus on encouraging agency heads to form a view on the current risk culture in their agencies, identify desirable changes to that risk culture, and take steps to address those changes.
Appendix one - Response from agencies
Appendix three - About the audit
Appendix four - Performance auditing
Parliamentary reference - Report number #298 - released 23 April 2018
Detecting and responding to cyber security incidents
A report released today by the Auditor-General for New South Wales, Margaret Crawford, found there is no whole-of-government capability to detect and respond effectively to cyber security incidents. There is very limited sharing of information on incidents amongst agencies, and some agencies have poor detection and response practices and procedures.
The NSW Government relies on digital technology to deliver services, organise and store information, manage business processes, and control critical infrastructure. The increasing global interconnectivity between computer networks has dramatically increased the risk of cyber security incidents. Such incidents can harm government service delivery and may include the theft of information, denial of access to critical technology, or even the hijacking of systems for profit or malicious intent.
This audit examined cyber security incident detection and response in the NSW public sector. It focused on the role of the Department of Finance, Services and Innovation (DFSI), which oversees the Information Security Community of Practice, the Information Security Event Reporting Protocol, and the Digital Information Security Policy (the Policy).
The audit also examined ten case study agencies to develop a perspective on how they detect and respond to incidents. We chose agencies that are collectively responsible for personal data, critical infrastructure, financial information and intellectual property.
Some of our case study agencies had strong processes for detection and response to cyber security incidents but others had a low capability to detect and respond in a timely way.
Most agencies have access to an automated tool for analysing logs generated by their IT systems. However, coverage of these tools varies. Some agencies do not have an automated tool and only review logs periodically or on an ad hoc basis, meaning they are less likely to detect incidents.
Few agencies have contractual arrangements in place for IT service providers to report incidents to them. If a service provider elects to not report an incident, it will delay the agency’s response and may result in increased damage.
Most case study agencies had procedures for responding to incidents, although some lack guidance on who to notify and when. Some agencies do not have response procedures, limiting their ability to minimise the business damage that may flow from a cyber security incident. Few agencies could demonstrate that they have trained their staff on either incident detection or response procedures and could provide little information on the role requirements and responsibilities of their staff in doing so.
Most agencies’ incident procedures contain limited information on how to report an incident, who to report it to, when this should occur and what information should be provided. None of our case study agencies’ procedures mentioned reporting to DFSI, highlighting that even though reporting is mandatory for most agencies their procedures do not require it.
Case study agencies provided little evidence to indicate they are learning from incidents, meaning that opportunities to better manage future incidents may be lost.
Recommendations
The Department of Finance, Services and Innovation should:
- assist agencies by providing:
- better practice guidelines for incident detection, response and reporting to help agencies develop their own practices and procedures
- training and awareness programs, including tailored programs for a range of audiences such as cyber professionals, finance staff, and audit and risk committees
- role requirements and responsibilities for cyber security across government, relevant to size and complexity of each agency
- a support model for agencies that have limited detection and response capabilities
- revise the Digital Information Security Policy and Information Security Event Reporting Protocol by
- clarifying what security incidents must be reported to DFSI and when
- extending mandatory reporting requirements to those NSW Government agencies not currently covered by the policy and protocol, including State owned corporations.
DFSI lacks a clear mandate or capability to provide effective detection and response support to agencies, and there is limited sharing of information on cyber security incidents.
DFSI does not currently have a clear mandate and the necessary resources and systems to detect, receive, share and respond to cyber security incidents across the NSW public sector. It does not have a clear mandate to assess whether agencies have an acceptable detection and response capability. It is aware of deficiencies in agencies and across whole‑of‑government, and has begun to conduct research into this capability.
Intelligence gathering across the public sector is also limited, meaning agencies may not respond to threats in a timely manner. DFSI has not allocated resources for gathering of threat intelligence and communicating it across government, although it has begun to build this capacity.
Incident reporting to DFSI is mandatory for most agencies, however, most of our case study agencies do not report incidents to DFSI, reducing the likelihood of containing an incident if it spreads to other agencies. When incidents have been reported, DFSI has not provided dedicated resources to assess them and coordinate the public sector’s response. There are currently no formal requirements for DFSI to respond to incidents and no guidance on what it is meant to do if an incident is reported. The lack of central coordination in incident response risks delays and increased damage to multiple agencies.
DFSI's reporting protocol is weak and does not clearly specify what agencies should report and when. This makes agencies less likely to report incidents. The lack of a standard format for incident reporting and a consistent method for assessing an incident, including the level of risk associated with it, also make it difficult for DFSI to determine an appropriate response.
There are limited avenues for sharing information amongst agencies after incidents have been resolved, meaning the public sector may be losing valuable opportunities to improve its protection and response.
Recommendations
The Department of Finance, Services and Innovation should:
- develop whole‑of‑government procedure, protocol and supporting systems to effectively share reported threats and respond to cyber security incidents impacting multiple agencies, including follow-up and communicating lessons learnt
- develop a means by which agencies can report incidents in a more effective manner, such as a secure online template, that allows for early warnings and standardised details of incidents and remedial advice
- enhance NSW public sector threat intelligence gathering and sharing including formal links with Australian Government security agencies, other states and the private sector
- direct agencies to include standard clauses in contracts requiring IT service providers report all cyber security incidents within a reasonable timeframe
- provide assurance that agencies have appropriate reporting procedures and report to DFSI as required by the policy and protocol by:
- extending the attestation requirement within the DISP to cover procedures and reporting
- reviewing a sample of agencies' incident reporting procedures each year.
Appendix one - Response from agency
Appendix two - ISMS maturity model
Appendix three - About the audit
Appendix four - Performance auditing
Parliamentary reference - Report number #297 - released 2 March 2018
