Attainment Scotland Fund evaluation: interim report (years 1 and 2)

9. Interventions – long term outcomes

9.1. This chapter explores in more detail the extent to which different types of interventions were making progress towards the longer term outcome of improving attainment and Health and Wellbeing, and why.

9.2. Specifically, it looks at what evidence was used to track progress, what this evidence said and what factors stakeholders thought contributed/hindered improvements in attainment and Health and Wellbeing.

9.3. This chapter therefore focuses on the data and measures employed by local authorities and schools, i.e. how they decided to monitor long term progress towards raising attainment and closing the poverty related attainment gap. Later in Chapter 11, the report shows statistics around attainment and how this compares across authorities and SIMD.

Chapter Highlights – Outcomes of interventions

• Overall, stakeholders felt that it was too early to comment on long term outcomes, but that initial indications were good. Many spoke of seeing an upward trend in indicators, and positive evidence from small scale projects.
• All authorities were collecting a range of evidence to measure long term outcomes. Each authority chose their own measures. The depth and breadth of data varied across authorities.
• Evidence of progress was clearer for the primary programme than the secondary programme, across both Challenge Authorities and the Schools programme. Many felt that it was too early to see any progress yet in secondary schools.
• Progress towards Literacy and Health and Wellbeing outcomes was generally well described.
• There was less evidence of planned activity in relation to Numeracy, largely due to a prioritisation of Literacy and Health and Wellbeing.
• A wide range of stakeholders stressed that it was important to recognise that a wide range of factors, other than school interventions, were likely to affect attainment.

What evidence did schools and authorities use to assess the extent to which interventions are improving attainment and Health and Wellbeing?

9.4. Challenge Authorities were asked to report on their measures of long-term outcomes of the Attainment Scotland Fund in their progress reports submitted bi-annually.

9.5. Authorities indicated that they were collecting a range of evidence to measure long term outcomes. Each authority chose their own measures, and as such there is no consistent picture across the board. The depth and breadth of data varies across authorities.

9.6. At least one of the following data sources was collected by Challenge Authorities to assess long term progress in attainment and Health and Wellbeing:

• Attainment data – the most commonly reported measure of long-term outcomes was attainment data. Authorities reported measuring attainment through a variety of measures; mostly, standardised Numeracy and Literacy assessments such as Performance Indicators in Primary schools ( PIPs), York Assessment of Reading Comprehension ( YARC) or New Group Reading Test ( NGRT). Many authorities mentioned teacher judgements and the tracking of Curriculum for Excellence levels. Some authorities explicitly stated that they were comparing SIMD data to track progress towards closing the poverty related attainment gap.
• Health and Wellbeing measures – measures of long-term outcomes in Health and Wellbeing included self-evaluation at school level, health trend data, exclusion/attendance rates, and assessments such as Boxall profiling or tracking of the SHANARRI (Safe, Healthy, Achieving, Nurtured, Active, Respected, responsible, Included) indicators. Overall, less in-depth information was provided about measures of long-term changes in Health and Wellbeing compared to changes in attainment.
• Research collaborations – a few authorities stated that medium and long-term outcomes were being assessed through research conducted by collaborators within the higher education sector.

What does local evidence show on the extent to which each of the different types of interventions contributed to improving attainment and Health and Wellbeing?

Ability to comment on long term outcomes

9.7. Many local authority officers (interviewed in the qualitative research) felt that it was still too early to comment on longer term outcomes, but that initial indications were good. Many spoke of seeing an upward trend in indicators and positive evidence from small scale projects.

9.8. Some local authorities interviewed were beginning to see the gap between the most and the least deprived decrease, but stressed that this was based on just two years of data. Views on progress were mixed. While some said that they had surpassed their targets, others felt that progress was not happening quickly enough, or that some targets were 'too big a reach'.

"Turning round kids' education is like turning round a huge slow ship. We are talking about deeper learning and understanding. You don't just turn a switch and it happens. It takes time" (Headteacher, Challenge Authority, Secondary School, Qualitative Research – Year 2)

9.9. At a school level, 77% of headteachers in Year 1 and, similarly, 78% in Year 2 stated that they had seen an improvement in closing the poverty related gap in Literacy attainment, Numeracy attainment or Health and Wellbeing in their school as a consequence of the Attainment Scotland Fund.

9.10. On the other hand, teachers who took part in the qualitative research cautioned that it was too early to comment on long term progress but many were hopeful that they would see positive outcomes in the longer term. Teachers in secondary schools in particular highlighted that it was very early days for their work supported through the Attainment Scotland Fund, and were less likely to be able to comment on outcomes in any detail.

Outcomes across a range of areas

9.11. This section explores teachers' views on progress towards outcomes around Literacy, Numeracy and Health and Wellbeing. This is based on the evidence collected through the qualitative research. A full detailed analysis of the qualitative research can be found in Chapter 6 of Appendix A. Below is a summary of key findings.

9.12. Overall, teachers and local authority officers were generally more able to comment on outcomes around Literacy than Numeracy. There was less evidence on outcomes in relation to Numeracy, largely because work in this area had started later.

9.13. Progress towards Health and Wellbeing was highlighted as challenging to measure, but both teachers and local authority officers were able to point to signs of positive progress.

Impact on Literacy

9.14. Teachers commenting on outcomes in qualitative interviews reported that they had seen an improvement in Literacy attainment through evidence from standardised assessments and reading scores, as well as observations of increasing Literacy skills.

9.15. Most of the evidence related to younger age groups, i.e. those part of the 'primary programme'. As seen in previous chapters, the secondary programme was only introduced in Year 2; and hence data and evidence on impact was more limited.

9.16. There was variation in the way that schools measured progress. Some examples collected through the qualitative research include:

Example: "In one school, baseline testing for P3, 4 and 5 showed an improvement of 25% in Literacy. The school had delivered interventions to target groups and had 'control groups' in order to better measure the impact of the intervention. It also benchmarked itself against other schools".

Example: "One school has used the 'Read to Self' approach with P5 pupils. It began to see tremendous increases in reading age over a very short space of time. It was rolled out to the whole school, and now some pupils are making 36 months progress over 12 months".

[Qualitative Research – Interviews with Headteachers / Teachers – Year 2]

Impact on Numeracy

9.17. Both at a local authority level and school level there was less mention of progress in relation to Numeracy.

9.18. A few authorities stated that they had started work around Numeracy later in the programme, and as such there was limited evidence to report on.

Example: In one school, through the Number Talks programme teachers are seeing that pupils have an increased ability to talk about maths, and to talk about their strategies for answering questions and problem solving.

[Qualitative Research – Interviews with Headteachers / Teachers – Year 2]

9.19. While most teachers interviewed in the qualitative research felt that their Numeracy approaches were going well, they also had less evidence about outcomes in this area. As with Literacy, progress in Numeracy was reported in varying ways.

Impact on Health and Wellbeing

9.20. Authorities and schools found progress around Health and Wellbeing harder to measure. However, some saw positive outcomes from individual interventions.

9.21. In the headteacher survey (Year 2), respondents discussed Health and Wellbeing initiatives leading to positive outcomes such as children developing 'emotional Literacy' and becoming more ready to learn. Headteachers reported that a focus on Health and Wellbeing was important for increasing pupils' readiness for the transition between primary and secondary schools.

9.22. Ways of measuring progress varied by local authority and school. Reported progress in Health and Wellbeing included, fewer exclusions from schools, better punctuality; improvements in behaviour, and increased resilience or improvements in relation to SHANARRI indicators. Some examples of the changes reported by teachers who participated in the qualitative research are outlined:

Example: In one school, teachers evaluate Health and Wellbeing three times a year using SHANARRI indicators. Pupils self-evaluate using traffic lights to indicate how they are feeling each day. Teachers pick up on any amber or red lights, and meet with parents to address any issues. Teachers have noticed a change from red to amber, which is going in the right direction.

Example: In one school, the nurture group had helped some pupils return to the main class. They were now exploring how they transition pupils back into the main class full time.

Example: One nursery was beginning to see changes as children move into primary one. The support that they had provided to parents was already having a positive impact, for example in relation to regular routines and healthy eating.

[Qualitative Research – Interviews with Headteachers / Teachers – Year 2]

Why do school and local authority stakeholders think certain interventions worked well (or not) in improving attainment and Health and Wellbeing?

9.23. The majority of headteachers (77% in Year 1 and 78% in Year 2) indicated that there had been improvement in attainment and Health and Wellbeing as a result of the fund. Furthermore, nearly all headteachers (97% in Year 2), expected to see an improvement in attainment and Health and Wellbeing in the coming five years as a result of the fund.

9.24. Those who indicated that they had seen improvements or expected to see improvements as a result of the funding provided information about the factors that were helping interventions to succeed. The responses could be grouped into five key themes, shown in Figure 9.1.

Figure 9.1: Key factors helping interventions to succeed, headteachers survey

9.25. The most commonly perceived barriers by headteachers to raise attainment and close the poverty related attainment gap were:

Figure 9.2: Key barriers to achieving long term outcomes, headteachers survey

9.26. There were also a few less frequently mentioned barriers. A few respondents indicated that there were issues around insufficient funding, resources and time. For example, they reported that there was a lack of funding for specific projects. A few respondents also discussed local authority budget cuts. In Year 2, respondents reported that there were difficulties in finding appropriate space and accessing services, particularly for those in rural areas.