Evaluation of the Compliance and Quality of Biodiversity Duty Reports 2015

2 Method

This chapter outlines the approach taken to the study, providing a list of the tasks carried out alongside more detailed information on particular activities. These included desk based research such as developing the evaluation matrix to provide the structure for the evaluation, reviewing the identified biodiversity duty reports, organising an internet based survey of public bodies, and holding telephone interviews with a sample of public bodies. The chapter also describes the approach taken to identifying key steps from the biodiversity strategy that are relevant to public bodies and subsequently matching activities reported by public bodies to these steps. The assumptions, limitations and caveats relating to the method are also discussed. Limitations relate to identifying all the relevant public bodies and appropriate contact details, whilst caveats include the classification of public bodies, and the receipt of multiple survey responses from the same public body. The chapter also notes that for a number of reasons, this study is likely to have underestimated the contribution made by public bodies towards the key steps in Scotland's biodiversity strategy.

2.1 Overview

The approach to the study combined desk based research (e.g. searching for published biodiversity duty reports) with public body consultation through an internet based survey and telephone interviews. The following sections provide more detail on the method followed as well as the assumptions, limitations and caveats.

2.2 Approach

2.2.1 Outline of tasks

The study was broken down into six tasks including:

• Task 1: project initiation and inception meeting;
• Task 2: evaluating reports published by public bodies. This included developing an evaluation matrix, reviewing the reports against the evaluation questions, and identifying good practice in terms of reporting and activities carried out. It also involved engagement with public bodies including an internet based survey of public bodies and telephone interviews;
• Task 3: identifying why reports were not submitted and how the template and guidance could be improved. This task drew on the responses from the internet survey as well as the telephone interviews;
• Task 4: assessing the contribution of the biodiversity duty towards the delivery of the '2020 Challenge for Scotland's Biodiversity';
• Task 5: producing and testing the revised template and guidance; and
• Task 6: reporting and presenting the findings.

The study outputs include a final report, a research summary, a guidance document and a revised template.

Further information on the main activities carried out as part of the above tasks is provided below.

2.2.2 Developing the evaluation matrix (part of Task 2)

The evaluation matrix provided the framework for the evaluation. It set out the areas to be covered, the evaluation questions, the types of data required to answer these questions (the indicators) and the information sources for the data. The evaluation matrix was developed from the version initially produced for the study proposal, taking into account the aims and objectives as well as the issues discussed at the start-up meeting. The final version of the matrix can be found in Annex 1.

2.2.3 Reviewing identified biodiversity duty reports (part of Task 2)

A spreadsheet was set up to record information from the review of the published biodiversity reports. Part of the structure of the spreadsheet aligned with the biodiversity duty reporting template as developed by the LBAP officer network ^[10] . It included space to record examples of the various biodiversity related activities undertaken by public bodies.

This study identified 53 biodiversity duty reports from internet research. An additional three reports were made known to the study team through links provided by public bodies in the online survey. All 56 reports were reviewed, with their details recorded in the spreadsheet ( Annex 2 lists the 56 reports identified).

The published reports were also reviewed for good practice in terms of (1) the activities reported and (2) their reporting structure and style.

To identify best practice activities, the reviewed reports were first categorised by type of public body (e.g. local authority, executive non-departmental public body). Several activities were then identified from each group of reports according to whether they were replicable (i.e. other public bodies in different locations could implement them) and had direct links to, and benefits for, biodiversity. It was not possible to consider biodiversity outcomes when selecting best practice examples since this is the first round of biodiversity duty reporting, so most reports did not provide any information on the results of implementing activities. It is anticipated that such information will be more readily available in future rounds. Grouping the reports by public body type ensured that a range of best practice activities were identified from public bodies with varying capabilities, resources and knowledge relating to biodiversity.

For structure and style, this process involved identifying the different ways in which information had been presented (e.g. tables, diagrams) as well as whether the report was easy to read and informative. Although a rather subjective process, this review provided useful information for the development of the guidance, in particular by providing public bodies with examples of how the report could be structured.

2.2.4 Internet based survey (part of Tasks 2 and 3)

An internet based survey was developed to enable all public bodies to provide their opinions on the biodiversity duty reporting process. The first batch of survey invites were sent out on 24 ^th February 2016, with reminders issued on 14 ^th March 2016. A second round of reminders was emailed on 23 ^rd March 2016. The survey was closed on 4 ^th April 2016, with 81 public bodies providing a response ^[11] . One additional response was received after the closure of the online survey, bringing the total number of responses to 82. However, a review of the comments provided by respondents indicated that one public body that had completed the survey was not actually in existence during the reporting period (2012-2014), and therefore did not have to provide a biodiversity duty report in January 2015. Excluding this public body and its response resulted in an overall response rate of 58% (81 responses out of 139 eligible public bodies).

The survey questions were designed to capture information on the biodiversity reports themselves (e.g. whether the report was standalone or part of another document), as well as the process (for example, whether the public body had written its own report or asked someone else to do it). The full set of survey questions are provided in Annex 3. Survey logic (e.g. automated question routing) was used to ensure that public bodies were only presented with questions relevant to their situation.

The Scottish Government's National Public Bodies Directory ^[12] was used as the starting point to identify public bodies to invite to complete the survey. Contact details for individuals were obtained from the internet where possible, or through telephoning the public body concerned. In total, this study determined that 139 public bodies were in existence between 2012 and 2014, and thus could be expected to have published a biodiversity duty report in 2015. The list of public bodies identified from the National Public Bodies Directory is provided as Annex 4.

The results of the survey are provided in the Section 3 (Key findings).

Emails were additionally sent to three newly formed public bodies to see if they had any views on the biodiversity duty reporting process. None of these organisations responded.

2.2.5 Interviews (part of Tasks 2 and 3)

The internet based survey included a question asking public bodies whether they would be willing to be contacted further to discuss biodiversity duty reporting. Of the survey responses received, 84% (68) indicated that they were happy to do so. Telephone interviews were held with a sample of these public bodies, including both those that had published a report and those that had not. In total, the study held 11 interviews.

The interviews were used to obtain more information on several areas, including:

• Good practice examples;
• Reasons why public bodies may not have published a biodiversity duty report; and
• Possible actions that the Scottish Government could take to assist public bodies with the production of reports in the future.

The interview questions are provided in Annex 5.

To obtain information on good practice examples, the study took a targeted approach to selecting individuals for interview. The public bodies that had published a report and were willing to be contacted further were matched to the good practice examples identified from the biodiversity reports. The public bodies were then classified by type, with invitations sent to five public bodies of different types with different example biodiversity actions. Where no response was received following a reminder email, additional public bodies were contacted. In total, five interviews were held with public bodies who had published a biodiversity duty report.

For public bodies who had not published a report, the study selected a random sample for interview. Six interviews were held with organisations which had not published a report.

2.2.6 Identifying relevant biodiversity targets (part of Task 4)

To assess the contribution that is being made by the biodiversity duty to delivery of the biodiversity strategy: 2020 Challenge for Scotland's biodiversity (Scottish Government, 2013), it was first necessary to identify the targets that are relevant to public bodies and their activities. Factors affecting a public body's ability to implement biodiversity related activities could include the amount of funding available, staffing levels and general awareness of biodiversity amongst staff. These factors could apply to any public body. However, there are two characteristics that are deemed to affect the range of biodiversity related activities that a public body could potentially carry out. These are:

• Whether the public body owns or is responsible for land and/or assets; and
• Whether the public body's main responsibilities are linked to or involve biodiversity.

Public bodies that own land, and whose main responsibilities are linked to or involve biodiversity, could potentially carry out a wider range of biodiversity related activities than those public bodies that do not. As an illustration, where a public body owns land it could ensure that the land was managed to provide suitable habitat for a particular species. This activity is not available to an organisation that does not own land or assets, irrespective of funding or staffing levels.

Table 2.1 provides a matrix that brings these two characteristics together. It illustrates that public bodies with land and whose main responsibilities are linked to biodiversity have the most opportunities in terms of being able to carry out a wide range of activities (subject to other constraints such as funding). In contrast, those public bodies that do not own land and whose main responsibilities do not relate to biodiversity have a smaller number of potential biodiversity related activities available to them.

This categorisation was used to help identify the biodiversity targets that may be relevant to the biodiversity duty. Public bodies with a small range of biodiversity activities available to them are expected to be able to contribute to fewer biodiversity targets than those public bodies that could potentially carry out many different types of activity.

Table 2.1: Matrix identifying how land ownership and main responsibilities affect the range of biodiversity related activities open to public bodies

		Ownership of land and/or assets
		Public body owns/is responsible for land and/or assets	Public body does not own/is not responsible for land and/or assets
Extent to which main responsibilities are linked to or involve biodiversity	Main responsibilities of public body are linked to/ involve biodiversity	Opportunity to carry out a wide range of activities for biodiversity	Opportunity to carry out a moderate range of activities for biodiversity
	Main responsibilities of public body are not linked to/do not involve biodiversity	Opportunity to carry out a moderate range of activities for biodiversity	Opportunity to carry out a small number of activities for biodiversity

Biodiversity targets that are likely to be relevant to some or all public bodies were identified from the biodiversity strategy: 2020 Challenge for Scotland's Biodiversity (Scottish Government, 2013). This strategy includes aims, outcomes and key steps which are linked to pressures affecting biodiversity. Annex 6 provides a list of the outcomes and key steps from the strategy. It then identifies the likely relevance of these key steps to public bodies in terms of whether public bodies are thought to have the opportunity to carry out activities that could contribute towards the step. Note that having the opportunity to carry out a relevant activity does not necessarily mean that a public body has the required funding, staffing levels or awareness to do so.

Three classifications were used to categorise the key steps:

• Minority: a limited number of public bodies could directly contribute to this step;
• Some: a moderate proportion of public bodies could contribute to this step; and
• Majority/all: almost all public bodies (with exceptions) could contribute to this step in some way.

These classifications were not defined in terms of a number of public bodies, since this could have led to the omission of some key steps because they were deemed not to be relevant to an arbitrary number of public bodies. Instead, the assessment process was relatively subjective and involved thinking about each individual step in terms of:

• Location: for some of the steps, actions could be taken anywhere. In contrast, others require activities relating to a particular habitat type (e.g. peatland). Such steps will only be relevant to those public bodies which have access to the habitat type concerned.
• Whether any relevant national level policies/plans are already in place: for steps relating to national government policies and plans, national level action is likely to be required (e.g. establishing network of marine protected areas). These steps are assumed to be beyond the remit of the majority of public bodies.
• Whether the step required the application of a particular method: some steps may require the application of a particular specialist technique or approach. Such steps may only be relevant to a small number of public bodies.

This process resulted in the following classifications for the 32 key steps provided in the biodiversity strategy:

• 12 steps are thought to relate to a minority of public bodies;
• 12 steps are believed to be relevant to a moderate proportion of public bodies; and
• 8 steps are assumed to be relevant to the majority of public bodies.

Thus, for the purposes of this study, 20 key steps were thought to be relevant to the majority of public bodies (See Annex 6 for the individual classifications). Whilst the remaining 12 steps are not unimportant for biodiversity overall, this study focused on the steps to which a high number of public bodies could actually contribute. This ensures that the assessment of the contribution that the Biodiversity Duty is making to the delivery of the "2020 Challenge for Scotland's Biodiversity" (Scottish Government, 2013) provides results of interest to the majority of public bodies, and not just those who operate in particular locations or carry out specific functions.

2.2.7 Linking biodiversity targets (part of Task 4)

In addition to the key steps outlined in the biodiversity strategy, this study also needed to consider the contribution of the biodiversity duty to the Six Big Steps for nature outlined in the route map ("Scotland's biodiversity - a route map to 2020"; Scottish Government, 2015) and the Aichi targets from the Convention on Biological Diversity ^[13] . To facilitate this process, the Six Big Steps for nature and the Aichi targets were matched to the 20 steps identified from the biodiversity strategy as being relevant to a majority/all or some of the public bodies.

2.2.8 Matching biodiversity actions to biodiversity targets (part of Task 4)

To determine how the biodiversity duty was contributing to Scotland's biodiversity targets, the information extracted from each of the 56 reviewed biodiversity duty reports was compared with the relevant key steps identified from the biodiversity strategy. Activities undertaken by public bodies were matched to the key steps. It is important to note that given the level of detail available, this was a relatively subjective process. Therefore, to help ensure consistency in matching the activities to the key steps, one member of the project team undertook this task. Where an activity reported by a public body appeared to be relevant to one of the 20 key steps highlighted, this activity was deemed to be contributing towards the biodiversity strategy. It is possible that with more information, some of the activities might be reclassified as not contributing (or vice versa) to the steps. However, the nature of the analysis meant that a high level approach had to be followed when matching the activities to the key steps.

Since the key steps had already been linked to the Six Big Steps for nature (in "Scotland's biodiversity - a route map to 2020") and the Aichi targets, it was then possible to identify whether the reported activities were contributing to these targets as well.

2.2.9 Revising the template and producing the guidance (Task 5)

The process of revising the template began with the existing template (produced by the LBAP officer network). It took into account the suggestions made by respondents to the internet based survey as well as the points raised during the telephone interviews. It also considered the need for the next round of biodiversity duty reports to make reference to the route map published in 2015 ("Scotland's biodiversity - route map to 2020").

As discussed above when identifying relevant biodiversity targets, different public bodies have different levels of opportunity when considering the types of biodiversity activity they can undertake. The guidance needs to be appropriate and applicable to all public bodies, whatever their assets or level of knowledge about biodiversity. Therefore, when identifying good practice examples for inclusion within the guidance, care was taken to select examples from different types of public body.

During the telephone interviews, a small number of public bodies were asked if they were interested in reviewing or trialling the revised template. Several public bodies indicated that they were happy to look through the revised guidance and were sent a draft for comment after the first version had been reviewed by the Scottish Government. Five public bodies provided comments. These were taken into account when finalising the guidance.

2.3 Assumptions, limitations and caveats

2.3.1 Assumptions

This study relied on engagement with public bodies to provide much of the information required to feed into the analysis. The main assumptions when running the internet based survey were that public bodies were aware of the need to produce a biodiversity duty report and that they had an opinion on the reporting process itself. This may not necessarily have been the case for a variety of reasons, for example, the survey invite may have been sent to the incorrect individual or the person who produced the report may have left the organisation. However, it is assumed that a sufficient number of respondents would have been involved in the production of a biodiversity duty report and so would be able to provide informed opinions on the process followed.

A further assumption it is important to acknowledge relates to the biodiversity duty reports themselves. When searching for biodiversity duty reports, if an organisation included a section on biodiversity within their annual report or other document, this was taken to be the biodiversity duty report. This section may not actually have been written to meet the statutory requirement of the WANE Act. However, for the purposes of this study, we have included these reports within the evaluation to ensure we take all approaches to reporting into account and do not miss any potential good practice examples.

2.3.2 Limitations of the approach

The main limitations relating to the survey of public bodies included identifying the relevant public bodies and obtaining the email addresses of the most appropriate individuals.

This study attempted to identify and contact all public bodies that were in existence between 2012 and 2014 and thus should have submitted a report in January 2015. The Scottish Government's National Public Bodies Directory was used to help identify public bodies. However, this list relates to the tax year 2015-16, so might exclude organisations that were in existence prior to April 2015 but are no longer classified as public bodies. Furthermore, it includes new public bodies that were not established in 2012 and thus are not relevant for this round of biodiversity duty reporting. Internet research was undertaken to try and determine which public bodies were relevant. However, it cannot be guaranteed that all relevant public bodies have been contacted.

For some public bodies, contact details for named individuals are readily accessible on the internet. Other organisations only publicise a generic email address for all enquiries. Where no email address or contact form was apparent, public bodies were telephoned to identify the most appropriate individual or department to whom the survey invite could be sent. Whilst it was not possible to telephone all the relevant public bodies due to time and resource constraints, efforts were made to ensure that the survey invites were directed to the correct department within an organisation. It should be acknowledged that not all public bodies contacted have biodiversity or environmental officers. In a few cases, it was difficult to identify the most appropriate individual or department for the survey invite.

2.3.3 Caveats when considering the findings

The main caveat is that the assessment undertaken by this study is likely to underestimate the contribution of the biodiversity duty to the delivery of Scotland's biodiversity strategy and other biodiversity targets (including the Six Big Steps for nature and the Aichi targets). Whilst the information provided in the existing biodiversity reports was screened against the key steps deemed as relevant from the biodiversity strategy, it should be remembered that:

• Time and resource constraints meant that it was not possible to go through each public body's report in detail and allocate all of their activities to a key step.
• Public bodies are unlikely to have reported all their biodiversity activities, since over a three year reporting period, these could be numerous.
• Public bodies which have not produced a biodiversity duty report may still have undertaken activities for biodiversity under the Nature Conservation (Scotland) Act 2004.

Another caveat relates to the different types of public body (see Table 2.2 and Annex 4). The largest group is the executive non-departmental public bodies ( NDPBs), followed by local authorities (note that public body types have been determined from the National Public Bodies Directory). The requirements of the WANE Act apply to all public bodies irrespective of their type or size. However, differences in the numbers of and response rates for the various public body types mean that care should be taken when analysing the results. Even if several public bodies appear to express the same viewpoint on an issue, it may not be appropriate to assume that this is the viewpoint of the majority. It could just be applicable to a particular type of public body. It should also be noted that public bodies may self-identify with a different category to the one to which they have been allocated in the table above (and on the Scottish Government website). Thus, the survey responses may indicate more or less of a particular type of public body than in Table 2.2.

Table 2.2: Types of public body

Type of public body	Number identified
Executive non-departmental public bodies ( NDPBs)	33
Local authorities	32
Health bodies	23
Other significant national bodies	14
Executive agencies	7
Non-ministerial departments	7
Commissioners and ombudsmen	6
Public corporations	6
Tribunals	6
Advisory non-departmental public bodies ( NDPBs)	5
All public bodies	139

Notes: table only includes those public bodies identified as being in existence between 2012 and 2014 (the three year reporting period) and thus expected to submit a biodiversity duty report in January 2015. The total of 139 counts the six Justices of the Peace Advisory Committees as one public body.

One further point to bear in mind when considering the survey responses is that there are a few cases where two public bodies work together and share their administration resources. The survey responses indicate that this is the situation for at least four Scottish public bodies (i.e. two separate incidences of shared administrative resources were identified). In these instances, it may be that the same individual or team effectively has the responsibility of producing the biodiversity duty report for two organisations. To help manage this issue, the survey asked respondents to state the name of the public body for whom they were responding. Including this question also ensured that any duplicate responses by the same public body could be combined. When calculating response rates, multiple responses for one public body have been counted as one response.