Scottish Transport Appraisal Guidance
6. POST APPRAISAL
6.1.1 Monitoring is the process of gathering and interpreting information on the performance of a project post-implementation. This process should be on-going and will usually take place in conjunction with other information gathering exercises being undertaken by a local authority or other organisation implementing an option.
6.1.2 The term ‘project’ is used in this section for any option which has been implemented, whether it is an individual project or programme comprising a series of projects. Similarly, at this stage the term ‘project manager’ is used to describe those who would undertake the Monitoring.
PROPOSED MONITORING PLAN
6.1.3 As part of a STAG study a Monitoring Plan should be developed to outline how Monitoring will be undertaken following implementation. This should be reported in the STAG Report. Further guidance is provided in the Technical Database. It is important to consider the scope of Monitoring activity during a STAG study. Failure to do so will make it difficult to assess the impacts of projects after their implementation. A Monitoring framework should therefore be established early in a STAG study to ensure the gathering of relevant and appropriate information.
6.1.4 The Monitoring Plan should form an integral part of the development and implementation of projects. Selecting measurable indicators of progress towards meeting the Transport Planning Objectives, performance against the STAG Criteria and evaluating the impacts of the project on established policy directives should be seen as a priority. Key performance indicators (KPIs) must, however, be set early in the development process.
6.1.5 Developing the Monitoring Plan at the same time as setting targets and indicators can help in defining more affordable Monitoring programmes.
PROJECT MONITORING
6.1.6 Effective Monitoring requires the regular analysis of the information being gathered in order to continuously review the performance of the project against the established Transport Planning Objectives and STAG Criteria and the impacts of the project on established policy directives. Used in this way, Monitoring should identify any areas of under-performance, and should also identify factors causing under-performance, thus allowing practitioners to implement appropriate changes at an early stage.
6.1.7 Monitoring performance is fundamentally important as it allows a measurement to be made of whether a project has been successfully implemented or not. The current focus in transport planning, and many other areas of government, is to measure success in terms of Best Value. This requires a measurement of outcomes from projects to be made rather than the outputs of an authority’s activity.
6.1.8 The levels of effort and expenditure required to monitor a project will vary. There are a range of factors which should be considered when determining the appropriate level of effort and expenditure for a particular project, including the level of resources available (both in terms of time and finances); the scale of the project; the degree of innovation of the project; and the degree of risk exposure associated with adverse outcomes and the quality/robustness of the Monitoring outcome.
6.1.9 Resource requirements associated with Monitoring should also be determined by the amount of information already available. It is important for practitioners to scan for information that may already be in the public domain. This would include Best Value reviews, Audit Commission reviews and modernising local government sources. It should be noted there are also ongoing traffic surveys and national data sources to draw upon.
PARTICIPATION AND CONSULTATION
6.1.10 Stakeholders should be consulted during the development and implementation of the Monitoring Plan. Support for the Monitoring Plan is essential as when agreed it will ensure performance can be effectively monitored and the integrity of a STAG study is maintained.
REPORTING
6.1.11 The project manager should develop a Monitoring Report that reflects the proposed Monitoring Plan developed as part of a STAG study. The details of this Monitoring Plan should be clearly described within the Monitoring Report.
6.1.12 The Monitoring Report will allow performance against objectives and indicators to be formally recorded. Monitoring periods by their very nature are required to be flexible and responsive to the type of information which becomes available between prescribed Monitoring intervals.
6.1.13 A large, technical, document is not appropriate for a Monitoring Report. Rather, a summary report in which key findings and trends are identified and displayed in a readable format is preferable. The use of charts and diagrams rather than paragraphs of text to convey relevant information should be used wherever appropriate. As part of the Monitoring Report an indication of the scope and timing of the Evaluation should be provided.
6.1.14 As the results of a Monitoring Plan are assessed, the detailed performance indicators and targets may need to be re-defined. The development of revised targets and performance indicators must be carefully considered and be compliant with SMART principles and be linked to the Transport Planning Objectives, STAG Criteria and established policy directives. In particular, such targets and performance indicators must continue to be achievable, yet challenging.
6.1.15 Depending on the Monitoring results, it might be necessary to consider whether a detailed Evaluation is warranted before making any major changes to the project.
6.1.16 The Monitoring Plan itself may need to be reviewed over time and modified according to the extent to which it is achieving reliable and cost effective results.
6.1.17 Additionally, the Monitoring Plan, detailed performance indicators and targets may need to be reviewed and updated to appropriately reflect any changes to established policy directives.
6.1.18 For the purposes of the STAG Report it will only be necessary to provide an indication of the proposed scope of the Monitoring to be undertaken and the identification of appropriate indicators to measure performance against the Transport Planning Objectives and STAG Criteria and the impact of the project on established policy directives.
Monitoring: Key Points
The term ‘Monitoring’ describes an ongoing process which has an important role in determining the success of a project in achieving established Transport Planning Objectives and measuring the performance of the project against the STAG Criteria and the impacts of the project on established policy directives. Monitoring includes:
- The development of a proposed Monitoring Plan, as part of a STAG study, to outline how Monitoring will be undertaken, post-implementation, and the scope of the Monitoring process;
- The development of challenging but achievable key performance indicators (KPIs) clearly linked to the Transport Planning Objectives, STAG Criteria and established policy directives;
- The collection, analysis and interpretation of data relating to any number of established indicators. The amount of effort and expenditure required should be appropriate to the scale and nature of the proposed intervention; and
- The development of a Monitoring Report to detail the extent to which a project is delivering value for money and achieving the objectives set.
6.2 EVALUATION
6.2.1 Evaluation is a specific post-implementation event designed to identify whether or not a project is performing as originally intended, whether, and to what extent, it is contributing to established policy directives and whether the implemented project continues to represent value for money. An Evaluation will use information gathered for Monitoring purposes but will also involve data gathering, analysis and detailed interpretation that is particular to the Evaluation itself.
6.2.2 Evaluation is always undertaken against indicators and targets derived from the Transport Planning Objectives, STAG Criteria and relevant established policy directives for a particular project and involves comparisons of the ‘do-minimum’ with actual outcomes. It is, therefore, important that, at the outset, Transport Planning Objectives are set within a framework that assists subsequent Evaluation.
PROPOSED EVALUATION PLAN
6.2.3 As part of a STAG study an Evaluation Plan should be developed to outline how Evaluation will be undertaken following implementation. It is expected the Evaluation Plan would consider the following:
- Process Evaluation – this is conducted at an early stage in the existence of a project and which is primarily concerned with how well the project has been implemented, this is also known as formative Evaluation; and
- Outcome Evaluation – this is conducted once the project has been in existence for a sufficient period to enable an examination to be undertaken of actual performance against identified targets.
6.2.4 During a STAG study, the term ‘practitioners’ was used to describe those planning, developing and appraising an option. At the Evaluation stage the term ‘project manager’ is considered more appropriate and would be used to describe the manager who would adopt the Evaluation Plan outlined in the STAG Report.
PROCESS EVALUATION
6.2.5 Process Evaluation is concerned with implementation, and it is therefore necessary to establish a usable number of performance indicators and measures relevant to what is expected during implementation. These may be thought of as tests of good implementation practice.
6.2.6 Process Evaluation is particularly useful in the early stages of implementation when there is scope for amending a project to make it more efficient or effective. The precise point in time for such an Evaluation has to be judged carefully to ensure that initial problems have been addressed and resources are not being used ineffectively where a project is performing poorly.
6.2.7 There can be particular value in the undertaking of Process Evaluation to highlight issues such as project selection and planning, the application and funding process, the way in which funds are allocated and the management of the project at national and local levels.
6.2.8 Process Evaluation should also highlight issues for the future Outcome Evaluation, including the extent to which the information being produced by the Monitoring process is likely to be adequate for subsequent Outcome Evaluation.
OUTCOME EVALUATION
6.2.9 Outcome Evaluation should look for clear and measurable outcomes from the project. The timing of an Outcome Evaluation needs to be carefully programmed. If undertaken too soon, final impacts may not have had time to ‘work through’, but if undertaken too late, resources will be wasted if the project is not efficient or effective.
6.2.10 Outcome Evaluations are intended to answer questions such as ‘what is the extent of the identified outcomes, and what were the costs of achieving this?’ and, where comparisons can be made with similar projects, ‘do these resources and outcomes together represent value for money?’.
6.2.11 The process used in an Outcome Evaluation may be set out as a series of sequential steps, as follows:
- Definition of scope and purpose;
- Project rationale;
- Aims and objectives;
- Measures and indicators;
- Base case for comparison;
- Analysis and interpretation; and
- Reporting and recommendations.
PARTICIPATION AND CONSULTATION
6.2.12 Stakeholders should be consulted during the development and implementation of the Evaluation Plan. The support and acceptance of the Evaluation Plan by Stakeholders is essential to ensure the performance of the project can be effectively evaluated and the integrity of the process of STAG is maintained.
REPORTING
6.2.13 The project manager would be expected to prepare an Evaluation Report based on the Evaluation Plan developed as part of the STAG study. An Evaluation Report will be required to take into account outputs from both the process and Outcome Evaluations undertaken.
6.2.14 In summary, the Evaluation Report should show whether a project represents a good use of resources, whether value for money could be improved, and, if so, how best to achieve this.
6.2.15 Further detail on the specific reporting requirements expected from the Process Evaluation and Outcome Evaluations undertaken are provided in the Technical Database.
6.2.16 For the purposes of the STAG Report it will only be necessary to provide detail on the proposed Evaluation Plan, outlining the scope and timing of the Evaluation to be undertaken together with an Evaluation Summary Table to be used to report outturn performance and impacts following implementation.
Evaluation: Key Points
The term ‘Evaluation’ describes a detailed, one-off objective driven review or audit of a project’s performance and includes:
- The development of an Evaluation Plan as part of a STAG study to outline how Evaluation will be undertaken post-implementation;
- Process Evaluation. This is carried out early in the life of a project, before its full effects are known and concentrates on whether input (activity) and expected outcomes for a project are being/have been met;
- Outcome Evaluation. This is carried out once sufficient time has elapsed for the project to have delivered its principal outcomes, and assesses whether the Transport Planning Objectives have been achieved and the performance of the project against the STAG Criteria and the impacts of the project on established policy directives and whether this has been done effectively and efficiently; and
- The preparation and completion of an Evaluation Report, based on the outputs from the Process Evaluation and Outcome Evaluation undertaken.
