What the report is about
This audit examined a state-wide program to provide small-group tuition to students disadvantaged by the move to learning from home during 2020.
The audit assessed the design and implementation of the program.
What we found
The program design was based on research and data showing learning loss during 2020.
The department rapidly planned and developed the policy design and guidelines for schools.
Governance arrangements matured during program delivery.
The department changed the models for funding schools but did not clearly explain the reasons for doing so.
Government schools with over 900 students were disadvantaged by the funding model compared to smaller schools.
Guidelines, resources and professional learning helped schools implement the program.
Staff eligibility for the program was expanded after reported difficulties in recruiting qualified teachers in some areas.
Online tuition and third-party provider options were developed throughout the program.
There were issues with the quality and timeliness of data used to monitor school progress.
Evaluation arrangements were developed early in the program.
Data limitations mean the evaluation will not be able to fully assess all program objectives.
What we recommended
- Distributing funds between schools more equitably and improving communication of the funding methods.
- Clearer communication about the intended targeted group of students.
- Reviewing the time needed to administer the program.
- Improve support for educators other than qualified teachers.
- Offer the online tuition program to more schools.
- Analysis of the effects of learning from home during 2021 across equity groups and geographic areas.
- Working with universities to increase use of pre-service teachers in the program.
The report also identifies lessons learned for future programs.
- $337m in total program funding. $289 million for government schools and $31 million for non government schools
- 12 days to develop the policy and provide costings to Treasury
- 290,000 targeted students in government schools and 31,000 in non government schools
- 80% of schools were providing small group tuition by the target start date of Week 6, Term 1
- 2–4 months was the estimated student learning loss from the move to learning from home during 2020
- 7,600 tutors engaged in the program as at September 2021.
The NSW Government announced the COVID Intensive Learning Support Program on 10 November 2020, as part of the 2020–21 NSW Budget. The primary goal of the $337 million program was to deliver intensive small group tuition for students who were disadvantaged by the move to remote and/or flexible learning, helping to close the equity gap. It included:
- $306 million to provide small-group tuition for eligible students across every NSW Government primary, secondary and special purpose school
- $31.0 million for around 400 non-government schools to provide small-group tuition to students with the greatest levels of need.
The objective of this audit was to assess the effectiveness of the design and implementation of the COVID Intensive Learning Support Program (the program). To address this objective, the audit assessed whether the Department of Education (the department):
- effectively designed the program and supporting governance arrangements
- is effectively implementing the program.
This audit focuses on activities between October 2020 and August 2021, which aimed to address the first session of learning from home in New South Wales. From August to October 2021, students in many areas of New South Wales were learning from home again, but this second period has not been a focus of this audit. On 18 October 2021, the NSW Government announced the program would be extended into 2022.
The COVID Intensive Learning Support Program was effectively designed to help students catch up on learning loss due to the interruptions to schooling caused by COVID-19. The department rapidly stood up a taskforce to implement the program and then developed supporting governance arrangements during implementation.
Most students in New South Wales were required to learn from home for at least seven weeks during 2020 due to the impact of the Novel-Coronavirus (COVID-19). The department researched, analysed and advised government on several options to address the learning loss that resulted. It recommended small group tuition as the preferred option as it was supported by available evidence and could be rolled out at scale with speed. It identified risks of ensuring an adequate supply of educators and options to address those risks. Consistent with its analysis of where the impact of the learning loss was most severe, the department proposed to direct funding to schools with higher concentrations of students from the most disadvantaged backgrounds.
The department established a cross-functional taskforce to conduct detailed planning and support program implementation. Short timeframes meant the taskforce initially sought approval for key decisions from the program sponsor and existing oversight bodies on an as-needed basis before dedicated program governance arrangements were formalised. Once established, the governance body met regularly to oversee program delivery.
The COVID Intensive Learning Support Program is being effectively implemented. The department has refined the program during rollout to respond to risks, issues and feedback from schools. Issues with how schools enter data into department systems have affected the timeliness and accuracy of program monitoring information.
The department provided schools with guidelines, example models of delivery, systems to record student progress and professional learning. Around 80 per cent of schools had begun delivering tuition under the program by the target date. Schools reported issues with sourcing qualified teachers as a key reason they were unable to start the program by the expected date. In response, the department expanded the type of staff schools could employ, developed an online tuition program, and allowed schools to engage third-party providers to help schools that had difficulty finding qualified teachers for the program.
The department used existing systems to monitor school progress in implementing the program. This reduced the administrative burden on schools, but there were several issues with data quality and timeliness. The program included a mid-year review point to check whether schools were on track to spend their funding. This helped focus schools on ensuring funding would be spent and allowed for redistribution between schools.
The department considered program evaluation early in policy design and planning. It embedded an evaluator on the taskforce and expanded a key assessment program to help provide evidence of impact. A process and outcome evaluation is underway which will help inform future delivery. The evaluation will examine educational impacts for students participating in the program but it has not established methods to reliably assess the extent to which the program has met a goal to help 'close the equity gap' for students.
1. Key findings
The policy design phase considered evidence and weighed options to address learning loss due to COVID related disruptions
The department developed an understanding of the needs and models of small-group tuition during the policy planning stage. Small-group tuition was identified as an appropriate method to help students catch up on learning loss. Policy planning included analysis of data from check-in assessments, external research and models used in other jurisdictions. The benefits and risks of alternate options were discussed with Treasury prior to being presented to the Minister for endorsement.
Budget costings overestimated what each teacher could deliver, which required lowering the number of weeks each targeted student would receive support in the final model
Costings developed for the budget proposal underestimated the total costs of delivering the program. The costing estimated an hourly rate for teachers based on them being available to deliver tuition for 35 hours per week over the full 52 weeks of the year. This did not account for school holidays, the hours available for teaching in a school day or the time needed for administration. The costing also assumed an average group size of five students, which was at the upper limit of a recommended size of 2–5 students.
The initial costing was developed on a basis of delivering three 45-minute sessions over 40 weeks to around 313,300 students in the bottom quartile of socio-economic advantage (260,600 government students and 52,700 non-government students). The final program model lowered expectations to support students over 20 rather than 40 weeks (which was still in line with the evidence base).
Soon after the proposal was approved, the department changed how it targeted funding
After the budget announcement, the department changed how it targeted funds. It used a different index to identify socio-economic disadvantage in government schools (the Family Occupation and Education Index) to align with its needs-based funding model. It increased the number of targeted students in government schools from around 260,000 to 290,000. To balance the higher number of students, funding was reduced from $1109 to $983 per student.
The new method varied funding provided for students in the second lowest socio-economic quartile depending on the overall size of the school. Smaller schools received a higher proportion of funding for these students than larger schools. The model introduced inequitable outcomes for schools around a cut-off point of 900 students (around ten per cent of all schools). Once this was identified in January 2021, the department considered changing the model but did not proceed as it would have required reallocating funds from medium-sized to larger schools.
For non-government schools, the funding model was changed to target schools with more than 15 per cent of students in the lowest socio-educational advantage quartile. The funding rate was increased from $554 to $865 per student and the targeted number of students was reduced from around 52,700 to 33,400.
A cross-functional taskforce helped to implement the program rapidly, which was supported by governance arrangements finalised during implementation
During policy development, the department identified the need for a separate taskforce to run the program. It stood up a cross-functional taskforce as soon as program funding was approved. This recognised the complexity of the program and short timeframes. The policy team had done some of the initial planning about the taskforce structure and workstreams. But there was no existing standard governance model in the department for taskforces.
Key supporting infrastructure for the taskforce (e.g. scope, terms of references and program schedule) were prepared in draft form but not finalised before key decisions were made. This increased risks to accountability and successful program delivery. In the absence of a formal governance model, the taskforce initially sought approval on key matters from the program sponsor and existing governance bodies.
The final governance model gave the program dedicated senior executive oversight through a program steering committee. An initial advisory group was established in February 2021 but only met three times and was replaced by a steering committee. The steering committee first met on 18 March 2021 and on a regular basis thereafter to review progress and emerging risks.
Eighty per cent of schools started to implement the program by the expected date
Following the program announcement, the department worked rapidly to give schools program guidelines alongside their allocation. The guidelines were succinct and focused on helping schools to implement the program as early as possible but at least by week six of Term 1. It was important that schools could commence the program early as delaying the program could increase risks that students needing additional support would fall further behind.
The department used existing systems to monitor how many educators had been engaged for the program. From this data, around 80 per cent of schools had started the program by Week 6, Term 1. By the start of Term 2, 2021 most schools had begun to implement the program. Schools reported a main reason for delays was problems sourcing qualified teachers for the program.
In March 2021, the taskforce advertised an expression of interest for 26 Deputy Principals (in a part-time capacity) to support schools with implementation. It was required to scale back this model of support in response to concerns about the impact on school-based staff raised at the new Steering Committee's first meeting. This delayed the support team from being fully operational until May 2021 and increased pressure on other taskforce members to support schools with implementation.
Guidance materials did not directly specify the students who should receive support, which increased risks that schools would target funding towards other priorities
The department provided schools with an expected number of students to support based on their level of funding. Program advice differed slightly in suggesting how students should be prioritised. For example, from students 'most at need' to those 'falling behind in their learning' or students who 'will benefit the most'. Guidelines also suggested schools consider 'system-negotiated targets' when planning the program (e.g. increasing the proportion of students in the top two bands of NAPLAN).
Directors and schools we spoke with told us that the lowest performing students were already well supported by existing programs. Because of this, schools chose to target funding at students performing at an average level. We identified a small number of schools that reported explicitly targeting students to move into the top two bands of NAPLAN. This does not meet the intent of the program, which was to increase the achievement of students who were disadvantaged by the move to remote and/or flexible learning, helping to close the equity gap.
Data quality issues affected the accuracy of progress monitoring and reporting
Schools were given a choice of two systems to record the number of students being supported by the program. There were data quality and timeliness issues with both systems, which has meant the department did not have a complete and accurate view of progress during program rollout.
Both systems record the total number of students in small group tuition, which may include contributions from other funding sources. The systems can account for the session length and number of weeks of tuition, but the quality of this data is questionable. The department advised in October 2021 that it was currently assessing the impact of potential under or over reporting.
The department planned for program evaluation early and dedicated resources to this
Evaluation needs were considered during policy planning and program development. The taskforce included an Evaluator from November 2020, which helped it to consider evaluation needs during program planning and rollout. The department allocated funding to 'check-in' assessments in Term 4 of 2021. These assessments will help demonstrate the impact of the program for participating students.
The evaluation plan covers most essential elements and includes a process and outcome component. The evaluation will examine educational impacts for students participating in the program but it has not established methods to reliably assess the extent to which the program has met a goal to help 'close the equity gap' for students. It is also unclear how effectively it will cover unintended consequences of the program, for instance the impact of the program on the availability of casual teachers.
The impact of students' learning from home during 2021 and any implications are currently unknown
Schools across areas of Sydney were required to shift to learning from home from 12 July, this was expanded to all areas of the State on 16 August. Students returned to face-to-face learning from 18 October (Kindergarten, Years 1 and 12) and 25 October (all other years).
The department advised schools to continue delivering the program in a learning from home environment. The quality of this delivery will have differed depending on the skills of the educator in delivering lessons online, and the accessibility of students taking part. It will be important for the department's evaluation to consider how the program was affected during this period.
For many students, the learning from home period in 2021 was longer than that experienced in 2020. However, both schools and students had more experience in teaching and learning from home. It will be important for the department to establish the scale of any impacts from the 2021 learning from home period and use this to inform future program delivery settings.
By January 2022, to inform the next stage of the program the Department of Education should:
1. review and revise the program funding allocation model to improve the equity of distribution and transparently communicate the method in advice to schools
2. review and revise guidance to clearly communicate the intended target group of students and the level of flexibility schools have to target other groups of students
3. review and revise guidance on the time schools should allocate to administering the program (e.g. co-ordination, lesson preparation, assessment, teacher collaboration) based on feedback from schools and key stakeholder groups
4. review the experience of schools that used initial teacher education students, retired teachers, school learning support officers, educational paraprofessionals, or allied health professionals to deliver the program.
Use these findings to update program guidelines, example models of successful delivery and professional learning offerings
5. review the model and outcomes of the online tuition program based on feedback from participating schools, with a view to increasing its scale.
By March 2022, the Department of Education should:
6. analyse and report on the effects of learning from home during 2021 for students across different equity groups and geographic areas
7. work with university stakeholders to identify ways to increase use of initial teacher education students in the program and promote examples of successful models of delivery.
Lessons for future programs
Delivering the COVID Intensive Learning Support Program required rapid planning and implementation. For future programs, the department should consider:
1. having in place standard costing approaches, including hourly cost rates for frontline staff and allowances for program administration
2. seeking stakeholder feedback on program design at the earliest possible opportunity and throughout program delivery to allow for adjustments to be made
3. having in place model governance arrangements for program taskforces that balance rapid decision-making with appropriate oversight
4. including an evaluation resource or perspective during program planning to enable timely collection of data at appropriate points of the program
5. providing early and clear guidance on program objectives, potential models of delivery, intended target groups, and the level of flexibility to vary program implementation
6. using a range of methods to monitor on-the-ground implementation to identify trends, understand problem areas and respond with targeted support.
1.1 2020 COVID-19 pandemic and NSW schools
Emergence of COVID-19 and remote learning for students
The emergence of the Novel-Coronavirus (COVID-19) brought significant health and safety risks to the public from January 2020, with increasing transmission and safety risks during February and March. On 11 March 2020, the World Health Organisation declared COVID-19 a pandemic.
The NSW Government required social distancing at schools from 15 March 2020, and encouraged students to learn from home from 24 March 2020. Schools conducted online learning for approximately seven weeks across Terms 1 and 2. The department developed and deployed online materials and other resources for continuing education through remote learning.
Impacts of learning from home
The department offered optional 'check-in' student assessments following the return to school from August 2020. These assessments were used in place of NAPLAN, which was postponed due to the pandemic. Check-in assessments were run for Years 3, 5 and 9 and had high take-up. The results showed that students were 2–4 months behind on average compared to 2019 results.
Research reported by the Grattan Institute in June 2020 also identified that students from lower socio-economic backgrounds faced greater educational disadvantage as a result from the move to learning from home.1 The report recommended catch-up learning strategies focus intensively on disadvantaged students. The report found the level of impact depended on factors such as:
- home environment being conducive to learning
- access to digital devices and internet connectivity
- parental/other support for learning at home
- capabilities and motivation for independent learning
- adapting to new routines for learning
- well-being and good health.
COVID Intensive Learning Support Program
On 10 November 2020, the NSW Government announced the $337 million COVID Intensive Learning Support Program (the program) to provide for small group tuition for students whose learning had been affected by the disruptions caused by the COVID-19 pandemic.
The program was one of the NSW Government's COVID Economic Stimulus Measures as part of the 2020–21 Budget. An economic stimulus aim of the program was to employ around 5,500 additional staff including: accredited teachers, retired teachers, initial teacher education students, university academics and post-graduate university students with appropriate qualifications. Small group tuition was estimated to be provided to up to 290,000 students for both government and non-government schools.
Funding was allocated to three components, with school funding for the 2021 school year:
- $289 million for all government schools
- $31.0 million for select non-government schools
- $17.0 million for program administration for government schools.
1.2 Delivery of the program
COVID Intensive Learning Support Program Taskforce
The department established a taskforce to develop and deliver the program for government schools. The taskforce was established on 9 November 2020, a day before announcement of the program. Its responsibilities included:
- setting human resources and industrial relations protocols (e.g. pay rates and conditions)
- developing a recruitment pool of tutors
- determining funding allocations to schools
- developing guidelines, support materials and professional learning resources
- monitoring schools' implementation and reporting to senior governance committees
- data collection and processes for evaluation.
School responsibilities for program delivery
School responsibilities for implementing the program included:
- selecting appropriate students to receive small-group tuition
- identifying and employing tutors
- assisting and supervising tutors that are delivering the tuition
- monitoring student progress and communicating with parents or guardians
- reporting to the department on program activities and student progress.
Allocation of funding
The department provided funding to schools based on the number of students in each socio-economic quartile. For government schools, it used the Family Occupation and Education Index (FOEI) to identify the socio-economic background of students. For non-government schools, it used the Australian, Curriculum and Reporting Authority's socio-educational advantage model.
All government schools received funding under the program. The minimum amount of funding was $5,600 for schools with up to ten students. The largest funding allocation was $698,000.
Funding to non-government schools was allocated if schools had greater than 15 per cent of students in the lowest socio-educational advantage quartile. Funding was provided to around 390 non-government schools and ranged from $5,500 to $547,000.
1.3 About the audit
The objective of this audit was to assess the effectiveness of the design and implementation of the COVID Intensive Learning Support Program. The audit focused on:
- the design and planning of the program
- the implementation of the program (both centrally and within government schools)
- monitoring and evaluation activities.
The audit interviewed ten schools as case studies. These schools included a mix of school types (primary, secondary, central, schools for specific purposes), locations (metropolitan, inner regional, outer regional) and levels of funding (low, medium and high as a proportion of total students).
Fieldwork for this audit was conducted from June to August 2021. For this reason, audit observations concentrate on program set up and implementation during Terms 1 and 2. Students in many areas of New South Wales were required to learn from home for most of Term 3. The audit did not examine in detail how well small group tuition was delivered in this context.
2. Program design
This chapter considers how effectively the COVID Intensive Learning Support Program (the program) was designed and planned for implementation.
2.1 Policy development and costing
Program design considered the available evidence base
The department considered research and analysis of similar catch-up learning programs implemented in other jurisdictions in designing the program. The department's analysis included:
- reviewing different models of tuition (e.g. allocating vouchers to be used in the private tutoring market and the direct engagement and management of tutors)
- examining the size of groups, the length of sessions, and duration of the program
- assessing the effectiveness and risks of different options.
The department's policy development concluded that small-group tuition could help students catch-up on learning disruptions from the 2020 COVID lockdowns. It identified that students from the lowest socio-economic quartile were likely to have been affected the most by lockdowns.
The department advised that it did not consult all relevant stakeholders during policy planning due to the proposal being part of an upcoming budget submission. It spent 12 days developing the proposal before submission to Treasury alongside costings. Engaging with relevant stakeholders at the earliest opportunity is important to consider any unknown elements that might affect the success of a potential program.
The initial budget costing overestimated what could be delivered in school settings
The department used a 'bottom up' approach to determine the funding required to provide small-group tuition to each student in the lowest socio-economic quartile. Key parameters were:
- length and number of tutoring sessions
- number of students per tutoring session
- staffing cost of delivering tutoring sessions
- program administration and implementation costs.
The budget costing was developed on the basis of students receiving three 45-minute sessions per week for 40 weeks (90 hours of support). The costing was based on an average group size of five students. This was at the upper limit of the suggested model of between two to five students. Costing the program at the maximum group size did not consider that many schools would need to run smaller group sizes to account for their contexts. This meant the costing likely overestimated the total number of students that could be supported with a given amount of funding.
Hourly staffing costs were estimated from the annual salary of a full-time teacher employed for 35 hours per week over 52 weeks. This overestimated the capacity for face-to-face delivery as the program was to occur during school terms (around 40 instead of 52 weeks). This estimate led to a staffing cost of $62 per hour, which was at the lowest point in the final range of $62 to $89 per hour depending on staff qualifications and time of day for program delivery.
The initial budget costing identified that supporting 313,000 students would require 2,800 full time equivalent teaching staff. This implied that each full-time teacher could support 110 students (22 groups of five students). Each group would receive 2 hours and 15 minutes of support each week for 40 weeks. This equates to around 50 hours per week of face-to-face delivery for each teacher, without any time for administration. Working through the implications of the costing could have identified this was beyond what can be reasonably expected.
Soon after the program was approved by government, the department reviewed the initial costing to support an announcement of how many tutors it expected to employ. It applied a higher staffing cost of $82.50 per hour and allocated 25 per cent of tutor time for administration. This meant it needed to lower the estimated hours of support from 90 to 48.75 hours per student.
Program objectives aligned with data that students had fallen behind in expected progress
A primary goal of the program was to deliver intensive small group tuition for students with the greatest need, helping to close the equity gap for disadvantaged students. The motivation for the program was that students had been disadvantaged by the move to learning from home and that students from lower socio-economic backgrounds were relatively more affected.
The department used results from 'check-in' assessments to inform policy development. In the absence of NAPLAN tests, the department ran optional check-in assessments for students in Years 3, 5 and 9. It compared results from the check-in assessments with NAPLAN results from 2019 and found that student learning was approximately 2–4 months behind on average.
The department did not analyse data on whether students from disadvantaged cohorts (e.g. low socio-economic status, Aboriginal, or English as an Additional Language or Dialect (EAL/D) or geographic areas subject to different periods of learning from home were relatively more affected. It advised this was because the check-in assessments were optional.
Research reports had identified that disadvantaged students in the lowest socio-economic quartile were likely to have experienced a larger learning loss. Directors and schools we spoke with reported students who had poorer access to technology, internet connectivity, and negative family circumstances were affected more during this period.
2.2 Allocation of funding to schools
A new funding model changed how funding was distributed across schools
Policy planning, budgeting and initial advice to the Minister was based on funding government and non-government schools according to the number of students in the lowest socio-economic quartile according to the Australian Curriculum and Reporting Authority's index of socio-educational advantage. This was around 260,000 students in government schools and 52,700 students in non-government schools. Following the budget announcement, the department developed two new funding models to determine final allocations to government and non-government schools.
The new funding model for government schools was based on a different index of socio-economic disadvantage (the Family Occupation and Education Index). The department increased the total number of targeted students by 11 per cent (from around 260,000 to 290,000) and reduced the amount of funding per student by around 11 per cent (from $1109 to $983 per student). The model was approved on 3 December 2020 with advice on funding allocations sent to schools the following day to allow for initial planning.
The new funding model for non-government schools selected schools with more than 15 per cent of students in the lowest socio-educational advantage quartile (around 390 schools). The number of targeted students was reduced from around 52,700 to 33,400 and the funding rate per student was increased from $554 to $865 per student.
The rationale and implications of changing the funding models were not well explained in briefings to senior management or the Minister. This was important as it was a deviation from the research report used as a basis for policy development and the initial parameters approved by government. The department advised that its initial costing was only for the purpose of constructing an overall funding envelope and that the revised funding model was more consistent with its methodology for providing equity funding for government schools.
There was no explicit rationale for providing less funding per student to larger schools
The funding model for government schools considered the total size of the school when determining the number of students eligible for funding. This meant smaller schools received proportionally more funding than larger schools (Exhibit 1).
|School size||Proportion of quartile 1 (%)||Proportion of quartile 2 (%)|
|Less than 300 students||100||100|
|300 to 899 students||100||50|
|900 students and larger||75||--|
Source: Audit Office of NSW analysis of Department of Education data.
The design of the funding model meant that school allocations could vary significantly around the cut-off points, particularly for schools above 900 students. Exhibit 2 shows an example of how the model could lead to inequitable outcomes. School A receives twice as much funding as School B despite having the same number of students in the lowest two socio-economic quartiles.
|School A||School B|
|Number of students in quartile 1 (lowest)||250||250|
|Number of students in quartile 2||250||250|
|Number of students in quartile 3||200||200|
|Number of students in quartile 4 (highest)||199||201|
|Students eligible for funding||375||187.5|
Source: Audit Office of NSW analysis of Department of Education data.
The department considered revising the funding model but did not proceed with changes
Once schools received their funding allocations, a key stakeholder raised concerns regarding inequities for larger schools when compared to other schools in their area. Earlier consultation with stakeholder groups may have identified concerns with the proposed funding model before advice had been provided to all schools, which raised expectations for what schools would receive.
In January 2021, the department prepared a brief to the Minister proposing to revise the funding model. The proposed change would have reallocated $15.6 million from medium-sized schools (300–899 students) to larger schools (over 900 students), a reduction of 8.55 per cent for medium-sized schools. Following discussions with the Minister's office and other stakeholders, the department decided not to proceed with the proposed changes to the funding model.
Once the decision had been made to keep the existing funding model, the department analysed data for 181 schools with over 900 students. It concluded that these schools would not be disadvantaged because they received high amounts of annual funding and could access unspent funds from previous years. The analysis was flawed because it did not consider the relative size of the schools. Doing so would have shown that schools with fewer than 900 students receive more annual funding and had significantly more unspent funds on a per-student basis.
The department invited directors to report any schools that felt their allocation would not meet their needs and advised that it might have capacity to redistribute some funding before Semester 2. It did not explicitly identify that larger schools would be given priority and only 30 of the 181 schools with over 900 students received additional funding for Semester 2 through this process.
The department negotiated funding agreements with non-government school peak bodies
The program budget announcement included $31 million for non-government schools. The department negotiated with two non-government school peak associations to co-ordinate delivery for schools in their respective sectors. The program is being delivered through funding agreements, which outline the department's expectations for program delivery and reporting. It expects that the program will support around 33,400 students in non-government schools.
Funding agreements and guidelines were finalised on 26 February 2021. Negotiating the agreements and guidelines required consultation to determine eligible schools, give flexibility in delivering the program and agree the obligations on the associations. The additional time taken to finalise agreements meant eligible schools had less time to plan and implement the program.
Funding agreements required the peak associations to report back to the department only once, by 31 March 2022. The lack of reporting during program delivery meant the department would not know how effectively the program had been implemented until it was due to have been completed. The department advised it did not request any reporting during 2021 in recognition of the delayed start to the program and to reduce the administrative burden for the sector.
Non-government schools received 88 per cent of the estimated cost to deliver the program
The program funding model provided non-government schools with 88 per cent of the amount for each targeted student compared to government schools ($865 compared to $983). The department did not disclose that non-government schools would be allocated a proportional amount per student in its negotiations with non-government peak bodies.
Funding agreements with the non-government peak bodies identify the number of students that each school is required to support but not the number of weeks they should receive tuition. Program guidelines suggest 'effective practice' is between 12–20 weeks for each student. Non-government schools face similar staffing costs as government schools, so may need to alter the program to fit their funding. For example, a non-government school student is funded to receive 17.5 weeks of support compared to 20 weeks of support for a government school student.
The department advised that the non-government school sector is aware that New South Wales does not provide full funding to non-government schools. It reasoned that non-government schools can supplement funding through parental fees and other income sources, including Commonwealth Government support. This program targeted funding to around 390 non-government schools with the highest concentrations of socio-economic disadvantage. These schools have less capacity to raise parental fees and other income. The funding agreements also did not identify that non-government schools were expected to contribute additional funding to the program.
2.3 Program governance and risk management
Supporting governance and implementation arrangements were drafted but not formalised until after significant decisions had been taken
When the taskforce commenced, it did not have a defined governance model, scope or terms of reference. These are important program artefacts to clearly outline how the taskforce will operate, how issues will be escalated, how decisions will be made, and the interdependencies with other parts of the department. The taskforce initially operated under draft governance documents and reported to the program sponsor and existing oversight bodies. Relying on existing oversight bodies created a risk to getting timely approval on rapid decisions needed for this program as these bodies covered a wide range of different programs.
Following program announcement, the taskforce worked to rapidly implement the program. It developed essential elements such as draft program guidelines, systems to recruit tutors and funding allocations for schools. Taskforce members were selected based on their experience in relevant fields and worked in close cooperation to determine immediate priorities and actions. The taskforce included embedded business partners from different areas of the department to advise on alignment with departmental procedures and priorities.
In February 2021, the department formed a program advisory group to bring together key stakeholders while program governance arrangements were being finalised. The purpose of the group was to provide direction to the program, manage dependencies and resolve conflicts. The advisory group met three times from 4 to 18 February before it was disbanded. The advisory group was not a fit-for-purpose governance arrangement for rapid decision-making.
The department replaced the Advisory body with a Steering Committee which first met in mid-March 2021. The Terms of Reference for the Committee were accepted by the group in late-April. The Committee had appropriate representation from relevant areas of the department and defined its role as a decision-making body that provides high-level oversight of the program. It initially met fortnightly and then monthly to monitor progress, risks and issues.
The department identified and advised on key risks during policy development
The department briefed the Minister on a range of issues and risks during policy development. These included:
- establishing an appropriately resourced, cross-functional project taskforce
- engaging with stakeholders and educational peak organisations
- developing guidance, resources and professional learning to support implementation
- finalising funding models for government and non-government schools.
A range of actions were also identified for addressing risks of workforce supply, including:
- determining a process for schools engaging retired, on leave, casual and temporary teachers
- considering innovative solutions for difficult to staff areas (e.g. remote delivery)
- working with universities to engage student teachers
- considering industrial relations implications of different staffing models.
The policy area supported the transition to the taskforce following program announcement.
The taskforce initially managed key risks with formal risk planning occurring later
Soon after it was formed, the taskforce held a workshop to identify key risks in early-stage implementation. Risk assessments are important when beginning a new program to identify the range of risks and develop plans to mitigate those risks at the earliest stage possible.
The taskforce's initial approach was to manage risks through daily stand-up meetings for the whole taskforce and weekly meetings between each stream lead and the head of the taskforce. It actioned significant tasks such as determining school funding allocations and guidelines before a comprehensive risk assessment was completed. The department advised it had limited time for typical program planning activities but was aware of the risks of rapid implementation.
In mid-December 2020, specialised program management staff worked with the taskforce to develop a formal risk assessment. The outcome included a program risk register and issues register which have been monitored and updated during program delivery. Registers detailed actions to address the risk or issue with a traffic light rating for the status of the risk or issue.
The taskforce reviewed risks and issues throughout the program and updated the program risk and issues register to reflect these. Additional review was conducted in May 2021 through engagement of a consultant to review the governance and implementation of the program, a separate review by a consultant engaged by NSW Treasury, and a taskforce workshop.
The taskforce reported to three key stakeholders/governance bodies on implementation progress, risks and issues throughout the program:
- School Improvement Executive Committee – monthly from February 2021
- NSW Treasury – monthly from February 2021
- COVID ILSP Steering Committee – initially fortnightly then monthly from March 2021.
3. Program implementation
This chapter considers how effectively the COVID Intensive Learning Support Program was implemented over our period of review (Terms 1 and 2, 2021).
3.1 Support provided to schools
Early guidance helped schools prepare for implementing the program
The department developed guidance and provided it to schools within 24 days of the program being announced. Advice provided to schools on December 4 included guidelines, example delivery models, and advice on hiring staff. Providing materials at the end of Term 4, 2020 helped schools plan for implementing the program by the start of 2021.
Initial guidance was presented in the form of 'Six key steps'. The steps included confirming funding, viewing the program website and introductory presentation, identifying students suitable for the program, choosing a delivery model and recruiting educators. The department published delivery models covering a wide range of settings, which was useful to avoid one-size-fits-all guidance.
Guidelines were provided to help schools navigate different types of educators that could be employed for the program. As well as teachers, there were two other categories:
- educational paraprofessionals - have completed a Certificate III in Education Support, or equivalent (or be in the process of completing these studies)
- educators (non-teacher) - retired teachers without current teaching accreditation, educators from the university sector, and students enrolled in a teaching qualification.
Initial communication described educators (non-teacher) but not educational paraprofessionals. Schools we spoke to told us of initial confusion about how to engage paraprofessionals as these are not common positions in schools. This added complexity for school planning for the program.
The department provided schools with clear guidance on the expected number of students to support based on their funding allocation. For example, a funding allocation of one full time teacher was expected to support 100 students with three, 45-minute tutoring sessions over 20 weeks.
A central expression of interest campaign identified potential educators for the program
To assist schools in sourcing qualified teachers, the department ran a centralised Expression of Interest campaign through a third-party system. It centrally reviewed applicants' qualifications, experience, criminal history checks, working with children checks and two referee reports. Schools could then use this system to search for teachers who had expressed an interest in the program. Schools were still required to interview individual candidates for the program.
As at 1 February 2021, the department reported 7,300 registrations to be part of the program, including from 6,400 qualified teachers. Schools we spoke with told us they preferred to use existing teachers or find teachers through their personal networks rather than a central system. In June 2021, the department reported around 500 schools had not used the central system.
Advice to schools was not consistent on which students should take part in the program
The department's guidance to schools on how to prioritise students for support was not always directly linked to program aims. The primary aim of the program was to 'increase the achievement of students who were disadvantaged by the move to remote and/or flexible learning, helping to close the equity gap'. In the absence of system-wide data on the impact of learning disruption, it was important for the department to guide school decisions to help meet the overall program aims.
Schools were responsible for identifying individual students to take part in the program. Program advice differed in emphasis for how students should be prioritised.
- Allocation and use of funds - 'for students identified in need of literacy and numeracy support, in order to be successful in meeting curriculum requirements'.
- Effective practice - 'students most likely to benefit from this additional support are students who are falling behind in their learning, particularly in literacy and numeracy.
- Getting started - 'have we collected qualitative and quantitative data that identifies those students most at need to participate in small group tuition'.
- Implementation and progress monitoring - 'schools will determine which students will benefit the most from small group intensive learning support with priority given to literacy and numeracy improvement for students from low socio-economic backgrounds'.
Guidelines on 'getting started' asked schools whether they had considered system-negotiated targets when determining the desired learning outcomes for the program. Common system-negotiated targets were set for schools in late 2020. They included:
- increased proportion of students achieving expected growth in reading and numeracy
- increased proportion of students in the top two bands in reading and numeracy
- increased proportion of students in the top two or three bands in HSC course results.
While supporting students not on track to meet their expected growth is consistent with the aims of the program, it is less clear whether targeting students to reach the top bands of reading, numeracy or HSC course results is consistent with the program aims. We identified 14 schools that stated they had explicitly targeted support for students to move into the top two bands. This is likely an underestimate as not all schools detailed their targeting approach in the reporting system.
Most schools we spoke with told us they used other funding sources and existing structures to support the lowest-performing students. They used COVID ILSP funding to support students in the next level of performance or those who had not demonstrated expected progress.
Professional learning sessions helped schools to deliver the program
Schools could access on-demand professional learning sessions through the department's intranet. On-demand learning gave schools greater flexibility in choosing what materials to access and when to access them. Schools we spoke to were generally positive about the range, quality and appropriateness of professional learning resources provided by the program.
Professional learning sessions covered both basics and a range of learning contexts. From mid-Term 1, the program offered eight professional learning modules specific to the program, which included topics such as 'best practice in small group tuition'. An 'Expert Series' commenced on 19 April and ran throughout the rest of the school year. The taskforce consulted with a range of internal stakeholders to identify possible professional learning topics that could be tailored to the program.
The site provided regular program updates in the form of twice-weekly 'coffee catch-ups'. It was also used as a space where schools could ask questions, share resources, and discuss challenges. As at 22 September 2021, the department reported around 7,300 staff had registered on the platform and around 6,000 had accessed professional learning.
3.2 Monitoring and reporting
The department monitored progress using existing systems but there were issues with the quality of data
The department developed several methods to monitor the number of staff employed and number of students supported. These were based on existing systems to enable them to be rolled out quickly and reduce the administrative burden on schools. It is important to have access to timely and reliable monitoring information so that any program adjustments can be made early.
The department asked schools to track program expenditure and recruitment using the schools finance system. The program monitored the number of educators on a headcount rather than FTE basis. This may have not given a full account of progress against expectations. By 5 July 2021, the department reported around 6,600 educators were involved in the program.
The department asked schools to track the number of students in one of two systems. It needed to modify one system, which meant it did not have oversight until mid-Term 2. Once these data were available, the taskforce contacted schools that were under-reporting student numbers. The department reported 115,000 students were taking part in the program by 5 July 2021, noting that 608 schools had not yet recorded data. We identified anomalies in the data used to monitor and report on the number of participating students, which raise questions over its quality and usefulness for monitoring. The department advised in October 2021 that it was assessing the impact of potential under or over reporting.
Reporting the total number of students may overestimate the coverage of the program. Schools could combine other sources of funding to expand the reach of the program. Reporting total students also does not show total hours of support, which depends on the average session length (e.g. 30, 45 or 60 minutes) or number of weeks (e.g. five, ten, 20 weeks). Schools could use both systems to record session length and number of weeks, but data was not consistently recorded.
Schools were also asked to report on how they were implementing the program into the school reporting system. Program guidelines suggested schools update information every five weeks. Most schools we spoke to told us they did not update information this frequently. The department advised it had not analysed or evaluated this descriptive information during program delivery.
80 per cent of schools started the program within the expected timeframe
Schools were asked to start the program as soon as possible and at least by 1 March 2021. By that date, 80 per cent of schools had reported commencing the program. Starting later than this increased pressure to accelerate delivery in later terms and increased risks to targeted students falling further behind. It was important for the department to have sufficient resources in place to support schools as they approached the deadline to start the program.
In early March, the taskforce proposed to engage 28 part-time Deputy Principal positions to support schools to implement the program. An expression of interest was advertised but was withdrawn when the newly formed Steering Committee raised concerns on the impacts of diverting staff from school-based work. The taskforce developed a new model with fewer staff, focusing on online, rather than face-to-face support. The delay meant the support team was not fully operational until May 2021. This delay increased pressure on the rest of the taskforce or other areas of the department to support schools with program implementation during the first Term.
A portion of funds were reallocated between schools depending on capacity to spend
The department provided schools with funding in two equal instalments for each Semester. Schools were told they would be asked to return a portion of funding if they could not spend it all by the end of the year. This was to reduce the risk of program underspend.
In early Term 2, Directors met with principals to discuss their progress with spending. Directors we spoke with told us this helped to monitor schools and reinforce messages of good planning and budgeting. Two aspects of the process, however, meant it was not fully equitable or transparent.
Firstly, Directors played a role in either encouraging or discouraging schools from requesting additional funds. Some Directors we spoke with told their schools there was 'nothing to lose' (most of their schools requested additional funds). Other Directors made either no or few requests. One school we spoke with was unaware they could request additional funds.
Secondly, there were no criteria given for prioritising requests and Directors were not informed of the reasons why schools were successful or not. The department advised that it chose not to provide feedback to 'reduce administrative burden'.
Around 480 schools requested additional funding of around $13.4 million ($2.8 million was available to redistribute from 36 schools. The department prioritised funding to:
- 4 schools experiencing rapid population growth (funding was based on 2020 enrolments)
- 30 schools with more than 900 students (as the funding model resulted in some large schools receiving a smaller allocation than similar schools with less than 900 students)
- 265 schools with small allocations (fewer students or higher socio-economic background).
3.3 Adapting the program
The program widened staff eligibility to allow more schools to commence
The department expanded the type of staff eligible to run small group tuition in recognition that many schools were unable to recruit teachers, paraprofessionals or educators (non-teacher). During the first weeks of Term 1, Directors raised concerns that some schools had tried, but were unable to source educators. They asked whether schools could instead use School Learning Support Officers (SLSOs). The taskforce initially approved requests on a case-by-case basis.
On 12 March, the department requested approval from the Minister to widen the eligibility criteria for all schools to employ SLSOs. The brief noted that many SLSOs were trained in delivering evidence-based interventions, had good connections with the community and that the use of trained support staff was supported by the research. The brief was approved on 29 March.
Directors we spoke with told us that some schools in their networks would have been unable to spend all of the funding without being able to employ SLSOs. At 22 September 2021, the department reported that 10.7 per cent of educators in the program were SLSOs. It will be important for the evaluation to consider the effectiveness of models of delivery by schools that used SLSOs as part of the program and promote these for schools to consider in 2022.
The department expanded eligibility for Schools for Specific Purposes (SSPs) and schools with support classes to employ Allied Health Professionals under the program from June 2021. This was done in recognition of existing shortages of qualified teachers in special education and the benefits that Allied Health Professionals could provide to students with disability.
Several Directors we spoke with told us that SSPs in their networks found it difficult to make the initial program guidelines work in their setting. SSPs already have small class sizes, meaning the recommended groups of 2–5 may split the class in half. They also noted that many students require one-on-one support, meaning the groups of 2–5 are too large to be manageable.
An online tuition program was created as an alternative for outer regional or remote schools
The department developed an online tutoring option for schools in outer regional and remote areas who were unable to recruit educators for the program. It identified the possibility of online tuition early in program planning, but substantive planning did not occur until mid-Term 1. This meant only a relatively small number of schools have been able to access online tuition.
A pilot program for online delivery began with two schools at the beginning of Term 2 (Exhibit 3). The pilot program required ensuring schools were technically capable, developing learning materials and employing tutors. The program was supporting around 170 students from ten schools by Term 3. While the limited scale of the online tuition means it has not formed a large part of the program, the model has potential to support schools in remote areas in future years.
The online tuition program delivers small group learning through an online platform to schools who were unable to source educators for face-to-face delivery of the program.
The department sourced educators through an expression of interest process for existing department teachers. Educators deliver lessons from a central location using online learning tools.
The online tuition team provides training and support for students and staff taking part in the program. Educators track student progress using PLAN2, and provide information to schools on how to use this information.
The program delivers 3 x 1-hour lessons per week in groups of between 2–5 targeted students. The school provides line of sight supervision (can be anyone with a Working With Children Check).
The department initially charged $75 per session for the program, which was later increased to $90 per session to ensure the pricing adequately covered program overheads.
The program was rolled out in three phases, and supported ten schools from Term 3 onwards:
The option to use third-party tuition providers from Term 3 has had low take up
From the beginning of Term 3, schools could also use third-party tuition providers to run the program. The use of third-party providers was identified during policy development and program planning as a way to address workforce shortages. Substantive planning started in March 2021. A tender process to identify suitable providers was released in late April. This timeframe meant the option was not available to schools before the start of Term 3. As at 11 October, the department reported six schools were using third-party tuition providers.
3.4 Program evaluation
Program evaluation was considered early in program implementation
The department considered program evaluation needs from initial policy development and advice to government. It embedded a Senior Evaluator into the program taskforce in November 2020. This has helped the program to consider evaluation needs as key decisions were made. Additional evaluation resources were later added to the taskforce during mid-2021.
The Steering Committee was briefed on plans for evaluation at its first meeting in March 2021 and an Evaluation Reference Group was also created to provide oversight of the evaluation. The Reference Group first met in July 2021 to discuss the terms of reference for the group, draft evaluation plan and initial survey results.
As part of the evaluation, the taskforce distributed surveys to principals, staff delivering tuition and classroom teachers in Term 2. It received responses from around 900 principals/co-ordinators, 400 staff delivering tuition and 220 classroom teachers. The taskforce also conducted field visits to 18 schools in Term 2 with observations of small group tuition sessions. The surveys and field visits complemented the evaluation approach to assessing the impact on student outcomes.
A portion of program funding has been allocated to expanding ‘check-in’ assessments, which will provide a key source of data for program impact. Check-in assessments were run in 2020 in the absence of NAPLAN. They had high take-up with 88 per cent of students in year 3, 86 per cent of students in year 5 and 61 per cent of students in year 9 sitting the assessments. For 2021, the assessments will be expanded to cover students in years 3 to 9 and will be held in Term 4.
The evaluation plan covers essential elements but does not include employment objectives
The draft evaluation plan covers essential elements including a program logic, evaluation questions, evaluation methods and data sources. The evaluation focuses on impacts for participating students but does not cover cost effectiveness to the same degree.
The NSW Government Program Evaluation Guidelines detail three types of evaluation: process, outcome and economic. It notes that the most rigorous and effective program evaluations combine elements of all three. Substantial resources have been directed to the COVID ILSP and it is important for the department to know whether the program has provided good value for money.
The draft evaluation plan identifies a challenge of accurately identifying educator qualifications. Not having access to educator qualifications is a missed opportunity. From initiation, the program has recognised workforce supply constraints for teachers in particular settings. Because of this, it allowed for the employment of other workforces (e.g. paraprofessionals and School Learning Support Officers). It is important for the evaluation to consider program outcomes from using different workforces. This information would be useful to inform the design of future programs.
The evaluation plan will not be able to fully answer whether one of the primary aims of the program has been achieved. A primary aim is to 'increase the achievement of students who were disadvantaged by the move to remote and/or flexible learning, helping to close the equity gap'. The evaluation will examine whether 'the COVID intensive learning support program improve[d] the academic outcomes of samples of students who participated'. There may be multiple 'equity gaps' depending on student cohorts, subjects and categories of equity disadvantage (for example, Aboriginal students or students with English as an Additional Language or Dialect). The evaluation plan does not include baseline measures of 'equity gaps' or ways to assess whether the program has helped to close those gaps.
As part of the draft evaluation, the department analysed data on the equity backgrounds of students participating in the program in schools using the PLAN2 system. It found that Indigenous students, students with higher English language support needs and students from lower socio-economic backgrounds were more likely to be participating in the program than their peers. The analysis also found students taking part in the program had lower average check-in assessment results. This provides a level of confidence that most schools have appropriately targeted students for the program.
The department identified a target program benefit of 'creating employment and a boost for the economy'. The evaluation plan, however, does not include methods to reliably assess the degree to which it ‘create[d] employment’. Relying on the total number of staff in the program risks overstating its impact as educators may have been diverted from other programs, or from the casual workforce. A department survey in July 2021 reported that 63 per cent of responding schools had used existing staff for the program and 47 per cent had used casual teachers known to the school. The department advised that it does not consider it necessary or viable to conduct an evaluation of the economic impacts of the program as the primary objectives were educational.
The evaluation plan doesn't explicitly consider unintended consequences
The evaluation plan includes a question to examine challenges encountered by schools, staff, and students. It is not clear the extent to which this will address unintended consequences of the program. The NSW Government Program Evaluation Guidelines encourages evaluators to consider 'unintended consequences' for participants or stakeholders.
The potential impact of the program on casual teacher supply was noted as a risk in early policy and planning advice and program updates. A program update on 5 July 2021 reported that the risk that casual supply would be impacted had been realised, and existing shortages of casual teachers had been exacerbated by the program.
Directors and schools we spoke with told us they found it more challenging to employ casual teachers in 2021 than previous years. This was seen as especially difficult in regional areas. They attributed difficulties in 2021 were due to casual teachers being employed for the program.
Non-government school peak bodies must also evaluate the program in their schools
Of the $31.0 million total allocated for non-government schools, the department quarantined $1.6 million for evaluation activities. A brief to the Minister in late February 2021 noted that evaluation costs would support check-in assessments and that the Centre for Education Statistics and Evaluation (CESE) would conduct further evaluation activities with the non-government sector. The draft evaluation plan does not reference the non-government sector and we were not provided with evidence of other planned evaluation activities supported by CESE.
Funding agreements with non-government school peak bodies require them to provide the department with an evaluation of the program, including an evaluation of the effectiveness and impact of the program on student progress and academic performance outcomes. The program allocated $150,000 for each peak body to administer the program, which included the costs of evaluation. The department provided peak bodies with a template to assist with the evaluation. The evaluations are due to be completed by March 2022.
At mid-September 2021, the department advised that it had allocated around $680,000 to support check-in assessments for non-government schools, leaving a balance of $820,000 unspent in the $1.6 million evaluation fund.
Capturing lessons learned and adapting the program for 2022
The department's education programs and interventions are typically planned to run longer than one year. This program demonstrated the resources and attention needed to work through the challenges of making a large-scale program work across the different settings of New South Wales public schools without time to adapt over successive years.
The taskforce had originally identified in its program schedule a series of activities to occur in December 2021. These planned to capture lessons learned and hand over the project to business-as-usual teams. An extension to the program was announced by the NSW Government on 18 October 2021. This gives the department an opportunity to reflect on the experience of the program during 2021 and adapt it for delivery in 2022.
In addition to the department's processes, this report has identified lessons learned that the department should consider when designing and planning future programs (see Executive summary).
© Copyright reserved by the Audit Office of New South Wales. All rights reserved. No part of this publication may be reproduced without prior consent of the Audit Office of New South Wales. The Audit Office does not accept responsibility for loss or damage suffered by any person acting on or refraining from action as a result of any of this material.
Parliamentary reference - Report number #358 - released (15 December 2021).