5 Issues for future roll-out, monitoring and evaluation
The first four chapters focused on summarising key findings and learning from the evaluation of the first FNP test site in NHS Lothian, Scotland, looking back over the past two and a half years. In this final chapter, we look forward to the future development and roll-out of FNP in Scotland. Drawing on learning from the evaluation of FNP to date and on the views and suggestions of those interviewed for this evaluation, we suggest some areas that are likely to require further consideration, monitoring and/or evaluation as decisions are made about the future implementation of FNP in Scotland.
5.1 Monitoring and understanding implementation within FNP in Scotland
This evaluation was deliberately limited in scope - it was intended to assess implementation, explore the potential for FNP to contribute to key outcomes and identify lessons learned from the very first FNP cohort in Scotland. As FNP expands across Scotland it is essential that the information being generated by FNP teams, FNP sites and at a national level is collated and interrogated on an ongoing basis. This will ensure that the programme is implemented and delivered as intended and that it continues to develop and respond to local context. A national monitoring and evaluation strategy which supports shared learning and continuous improvement at national and local level for FNP would help ensure that this happens in a regular and systematic fashion. The monitoring and evaluation framework generated for this evaluation would provide an obvious starting point for development of any future national strategy. Such a strategy would be able to draw on the evidence routinely collected by sites about all client visits, once the national FNP database is fully functional. Additionally, the framework would provide a useful basis to consider where evidence is weakest, and therefore the outcomes (and assumed mechanisms to achieve these) that should be prioritised in future evaluation activity.
Specific issues a national monitoring and evaluation strategy could cover include:
- Identifying any variations or wider implementation issues relating to the extent to which fidelity to the FNP model is achieved across (or within) sites or any changes in adherence to fidelity over time
- Exploring the reasons for any such variations, how success can be replicated and how challenges can be/have been overcome ( e.g. identifying areas for improvement and areas that are improving).
- Assessing any variations in the apparent feasibility of delivering FNP in areas where pregnancy rates among the target population of mothers are high or low (either of which could create challenges around workloads and sustainability)
- Research around barriers and enablers to delivery of FNP to specific client groups - for example, women from minority ethnic groups, or women with substance misuse problems.
- Understanding the particular circumstances associated with/ contributing to better (and worse) outcomes, using this evidence to inform future delivery.
- Further research in the wider NHS and with other key stakeholders in order to better understand how FNP has been received by other services, and how working relations might be further improved.
5.2 Monitoring and benchmarking key outcomes
As we have emphasised, without an RCT it is not possible to conclusively establish what, if any, impact FNP is having in Scotland over and above that which might be expected from delivery of routine antenatal and postnatal care to young families. The FNP National Unit (Scotland) will therefore need to identify appropriate benchmarks against which to interpret quantitative findings for Scottish FNP clients. In the short-term, it may be sensible to focus on a smaller-number of client outcomes, rather than attempting to benchmark every possible FNP outcome. These could be selected on the basis of those known to have been observed in RCTs of the programme (see Chapter One and Ball et al, 2012), and/or on the basis of outcomes that are considered particularly key to policy objectives in Scotland. Monitoring outcomes within and across Scottish sites against agreed benchmarks might then become a key element of any national monitoring and evaluation strategy (as described above).
Decisions about what constitutes an appropriate benchmark for specific outcomes will need to take account of a variety of issues including:
- the feasibility and acceptability of FNP collecting (good) data in a systematic and timely manner
- the appropriateness of the comparison data (Are they available for mothers under 20?)
- the robustness of the comparison data (Are they based on the whole population or a sample? How reliable is it?)
- the level at which comparison data are available ( NHS Board? Scotland-wide? UK-wide?), and
- comparability of data with that collected for FNP (How do the questions used to collect comparison data compare with those used within FNP? What issues might any differences in how they were collected create?).
In addition to monitoring and benchmarking key outcomes in the short-term, the FNP National Unit (Scotland) may wish to plan for monitoring and evaluating longer-term outcomes among FNP clients in Scotland. The evaluation on which this report is based only followed clients to the point just before their graduation from the programme. Further research is required to establish the potential for FNP to contribute to clients longer-term outcomes, beyond their involvement in the programme itself. While the FNP approaches are consistent with practices employed with health inequalities or 'youth employment' ( e.g. asset-based approaches, use of motivational interviewing etc), there would seem to be a need to explore client and wider services' experiences of clients' transitions out of FNP, their ongoing contact with universal services and how such services can build on and sustain positive outcomes.
5.3 Understanding and future planning for FNP workloads
As discussed in Chapter Three, views and experiences of the workload associated with delivering FNP varied across the course of the first two and a half years of delivery, as well as varying within the NHS Lothian, Edinburgh FNP team. Assessing workloads during a pilot phase of a programme is difficult - any findings may not be completely representative in terms of likely future workloads once initial training is complete and Family Nurses are more familiar with the programme and their role.
In the evaluation conducted by Barnes et al (2011) of the first 10 sites in England, they found that the amount of additional time Family Nurses on average worked over their standard hours decreased over time  . This could possibly be due to Family Nurses being more efficient in delivery as they get to know the programme better. A recent study of the Family Nurse workforce in England found that two thirds of Family Nurses and nine in ten supervisors said they regularly worked longer hours than those indicated by their contracts (Robinson and Miller, 2013). Given these findings, and the divergent views and experiences reported by the NHS Lothian, Edinburgh FNP team, there is arguably a need for a more systematic review of FNP workloads and monitoring by local sites as the programme moves beyond a pilot phase in Scotland.
Findings from such a review could then feed in to further decisions and guidance on how to assess whether or not sites are ready to 'scale up' their FNP service to recruiting clients on a rolling basis (sometimes referred to as 'small scale expansion'). Decisions on this are likely to involve a number of complex issues, including:
- How well prepared the local area is in terms of leadership, and in terms of other services' responses to and understanding of FNP.
- How local information should be used to estimate the likely in- and out-flow of eligible women over an extended period of time
- How to move from client population estimates to a decision about how many Family Nurses and Supervisors may be needed to enable sites to offer the programme to all or most eligible women in their area
- Differences across Health Board/geographical areas - for example, delivering FNP to a client populations that are highly dispersed ( e.g. in remote and rural areas), and/or that include women who are highly transient ( e.g. gypsy travellers who are unlikely to stay in the area for the duration of the programme) or moderately so ( e.g. due to availability of housing stock)
- How to balance the FNP fidelity requirement to ensure Family Nurse continuity with inevitable constraints around what is a feasible caseload for a Family Nurse when supporting dispersed or transient groups as described above
- What impact the recruitment of Family Nurses might have for the local Midwifery and Health Visiting workforce (an issue highlighted in both Robinson and Miller, 2013 and Ball et al, 2012)
In order to ensure that decisions about 'scaling up' are being taken on a consistent and coherent basis, and the quality is maintained, further work at a national level to produce guidance on these specific areas may well be required.
5.4 Maintaining learning from the international evidence base
In taking decisions about the future development of FNP in Scotland, it is important that the programme continues to learn from the existing and emerging international evidence generated by FNP on what works, for which groups of clients, and with what outcomes.
As Ball et al (2012) note, the precise benefits shown in the US trials are often only apparent in the medium or longer term. Moreover, they note that while FNP is often reported as having 'improved pregnancy outcomes', in fact such improvements were not apparent across all of the outcomes measured by FNP. While FNP remains one of the few early years programmes internationally that shows well evidenced benefits, they therefore suggest that FNP commissioners and practitioners in the UK need to understand more of the detail of the FNP evidence to avoid disappointment or 'over-promising' about what the programme may achieve.
Further, in a Scottish context, while in the short term there will clearly be a focus on what the English RCT reveals about the scope for FNP to have short-term impacts in a UK context, it is also important not to lose sight of evidence from elsewhere. For example, FNP programmes are currently starting in Canada and Australia. Given their geographies, there may well be learning from these countries about how to structure and support Family Nurse teams in rural and remote areas which is particularly relevant to Scotland.
Finally, and consistent with FNP's ethos and commitment to evidence based practice, it is recommended that attention be given to the translation of evidence: as such, to ensure that implementation is informed by emerging evidence, there will be an ongoing need to distil lessons learned, reflect on the implications of these for FNP in Scotland, and disseminate these in a manner that is not only targeted at key stakeholders and but also tailored to meet their information needs.
This evaluation has demonstrated that it is possible to implement the FNP programme with fidelity in a Scottish context. While it has been unable to measure or demonstrate impact over and above that which might have been achieved through existing services, it has also provided evidence that the programme may plausibly achieve its intended long-term outcomes - in other words, that many of the key mechanisms within its theory of change appear to be working as intended. Finally, and perhaps most importantly, it has highlighted a number of implementation issues that, if taken into account in the planning and delivery of future sites, will further improve the chances of success of the programme.
Family Nurse home visit during toddlerhood
Family Nurse Partnership Graduation Event