Regional employment patterns in Scotland: statistics from the Annual Population Survey 2016

Annex C: Confidence Intervals

One of the benefits of the boosted data is more reliable estimates for local authority areas. Prior to the boost the reliability threshold in all areas was 6,000. This was to prevent unreliable data being used. Thresholds are calculated so that they are approximately equivalent to suppressing if the standard error of an estimate is greater than 20% of the estimate itself. With the boost, different areas have different thresholds as some areas have larger samples and more variability in results than others (see Table 1).

Table 1: Local authority area reliability thresholds

As survey results, these are subject to a degree of error and implied changes over the years may not be significant and instead be within a given error range. Confidence limits can be used to assess the range of values that the true value lies between. The web tables include 95% confidence limits for each indicator.

What does the 95% confidence limit mean?

If, for example, we have an APS estimate and confidence limit of 63% +/- 0.27%, this means that 19 times out of 20 we would expect the true rate to lie between 62.73% and 63.27%. Only in exceptional circumstances (1 in 20 times) would we expect the true rate to be outside the confidence interval around the APS estimate. Thus the smaller the confidence limits, the more reliable the estimate.

The confidence limits use a design factor of 1, which may not be likely in some cases but given the lack of further information an average design factor of 1 is assumed to be reasonable. Further information on estimating confidence intervals can be found in the LFS manuals.

Using confidence intervals to assess change (statistical significance).

Confidence intervals can be used to assess whether there has been a significant change between two estimates over time. The methodology for determining if a change is statistically significant is detailed in the Methodology Glossary on the Scottish Government web-site within the Tier 2 - Confidence Intervals document, available at:

http://www.scotland.gov.uk/Topics/Statistics/About/Methodology/Glossary

If the difference between two estimates is said to be statistically significant, it means that only in exception circumstances (1 in 20 times) would we expect the true difference to be not significant. It should be noted that statistical significance is a tool used to help detect real change in estimates; it does not say anything about the importance of the change, which needs to be assessed by the user of the statistics in question.