The 5 Step Approach to Evaluation: Designing and Evaluating Interventions to Reduce Reoffending

Background: The tricky business of measuring impact in a messy world

How the 5-step approach came to be.

How was the pack developed?

Who developed this pack?
This pack has been developed by Scottish Government researchers in Justice Analytical Services with the aim of promoting and supporting the effective evaluation of criminal justice interventions.

What's in the pack?
We describe a 5-step approach to designing and evaluating interventions and services. It includes comprehensive summaries of the reducing (re)offending evidence-base and subject-specific logic models to support practitioners working in the field of crime and desistance.

Is the 5-step approach being used in practice?
The approach described in this pack is already being used widely by services, interventions and funders including Third-Sector organisations, The Robertson Trust, the Scottish Prison Service and the Scottish Government .

A Scottish approach to evaluation

Co-production
Our approach to evaluation enables funders and service providers to work together in pursuit of their shared aims - to improve outcomes for service users and communities. The 5-step approach also engages with service users' views as a resource for evaluation rather than seeing users solely as an object to be measured. In fact, most complex social outcomes can ONLY be achieved if we make a distinctive, yet joined-up contribution over a sustained period of time.

Asset-based
The 5-step approach focuses on ways in which evaluation is possible for services of any size, rather than expecting all services to use an experimental evaluation method which may not be appropriate or possible for smaller, community-based organisations. The 5-step approach allows even the smallest service to demonstrate the contribution they are making to change.

An Improvement Culture
Evaluation enables improvement and even the most successful service can always be developed further. Furthermore, with the 5-step approach, evaluation is an on-going process, not something to be saved for last. This means that services can be continually improved in order to best meet the needs of their users.

What are impact evaluations/RCTs

What is an impact evaluation or RCT?
An impact evaluation is designed to answer the specific question 'did my project / service work?'

An impact evaluation or RCT is a much like a scientific experiment and is often considered the 'gold standard'. One group (the 'treatment' group) experience your intervention and one group (the control group) does not. You then compare the outcomes for both groups to see if your intervention made any difference. In other words, if you really want to know if you've made a difference, you need to know what would have happened if the same (or similar) users DIDN'T receive your service. This enables you to ATTRIBUTE changes in users to YOUR service rather than other factors like motivation, another programme or family influences.

The control group must either be selected completely at random or otherwise be very carefully selected to have very similar characteristics. Otherwise, you cannot be sure that any apparent differences in results at the end are not the result of differences that were already there at the start and therefore nothing to do with your intervention.

Cost benefit analysis

Some funders ask for a cost benefit analysis which is an economic assessment that places a monetary value on the costs and benefits of an intervention. It is another way to determine the value of an intervention and convince others that it has public value.

What is a cost-benefit analysis?

• CBA usually builds on a rigorous RCT and typically measures a wide range of outcomes.
• It usually measures the public benefits to society but may also consider benefits to individuals and families.
• It is both an art and a science especially when assigning monetary values to social benefits such as better parenting or securing accommodation .
• CBA allows for comparisons across interventions, policies, and other types of interventions.

What data do you need?

• Cost estimates may be based upon well-documented impacts (i.e. evidence-based interventions)
• Cost estimates may be based upon well-documented impacts and future projections upon these documented impacts
• Cost estimates may be based on undocumented assumptions that the intervention works and hypothetical projections or 'what if' analysis - but no hard data is available

Excellent guidance on CBA can be found here and Justice Analysts are happy to provide advice: http://whatworks.uwex.edu/attachment/whatworks_cost_benefit.pdfa

The difficulty with RCTs

You need a large sample
RCTs are only meaningful IF there is a large control group with very SIMILAR CHARACTERISTICS to the users (the counterfactual). Scotland is a relatively small nation and behaviour change projects often target small or localised populations, making them hard to carry out.

They can be expensive
Funding may be a barrier since RCTs may be expensive to run and therefore not cost-effective as a means of evaluating small-scale projects.

They can't tell you everything
RCTs can't tell you WHY something is effective (or ineffective) so learning anything about HOW a project worked is tricky using this method.

Do impact evaluations / RCTs even ask the right question? Contribution not attribution

Example - contribution to achieving outcomes

Like most social outcomes, reducing crime and reoffending are long term, complex goals and hard for any standalone service to achieve. For example, we know that many studies show that the most effective way to reduce reoffending is through a well-sequenced, holistic approach which can address multiple needs such as the provision of quality accommodation, positive relationships and recovery from drug abuse. The question then becomes….if these combined services achieve a reduction in reoffending, which service is responsible? The answer is, of course that all of them have a distinctive role in contributing towards achieving the outcome……so it follows that any evaluation of a single service should assess the extent of their particular contribution (defined by their own objectives). Impact evaluations (RCTs) put all the pressure on single services to 'prove' they have reduced reoffending rather than evaluate the contribution they are making.

An alternative to RCTs

A "middle ground" approach
Rather than carrying out a small RCT which might be impractical and would only deliver meaningless results, or unreliable anecdotal research we recommend that small-scale project organisers carry out a 5-step approach to evaluation. This is summarised in the following pages and detailed in the remainder of this pack.

This approach to evaluation is practical for projects of any size but does rely on providers having a clear sense of what they're hoping to achieve or change and how they're going to get there - a theory of change. For this reason, using the 5-step approach to evaluation, we must begin at the planning stage.

What is evaluation really for?

Although doing evaluation requires the use of techniques and tools, bear in mind that its overall purpose is to help you (re) design services, ask questions, gather evidence, interpret the evidence, communicate important information about your service and take informed decisions. In this sense, the ability to ask relevant questions and clearly communicate the answers at the right time to the right people are key skills in making evaluation useful.