Effective implementation of RBM requires that evaluation become an integral part of the RBM framework of an organization18. Also, there is an increasingly widespread understanding that effective ways of managing-for-results require both RBM-oriented measurement of results, as well as evaluative information generated by evaluations. This suggests that evaluation is an important tool in promoting sound RBM systems that promote managing for results, help institutionalize a culture of reflection and learning, and contribute to accountability.
Evaluation at UN-Habitat is governed by regulations and rules of the United Nations Secretariat, as put forth by the Secretary-General in 200019, and guided by the UN-Habitat Evaluation Policy20, which is in conformity with the Norms and Standards for Evaluation in the UN System developed by the United Nations Evaluation Group21, as well as the Revised UN-Habitat Evaluation Framework document22.
The UN-Habitat Evaluation Policy was adopted in 2013 to strengthen the evaluation function in UN-Habitat. The policy describes the institutional framework for the planning, conduct, management, reporting, follow-up and use of evaluations. The norms and standards as expressed in the policy and the UNEG Norms and Standards for Evaluation in the UN system, guide the practice and use of evaluation in UN-Habitat. The revised framework document specifies evaluation performance targets for UN-Habitat, identifies which projects require evaluation, and outlines responsibilities of project managers and the Evaluation Unit as well as guides on how to cost evaluations.
The purpose of the evaluation chapter of this Handbook is to provide staff members who are either tasked with managing and /or conducting evaluations with practical guidance on conducting evaluations. The chapter is also useful to those responsible for designing, managing and monitoring programmes and projects.
This section addresses the basic questions of evaluation: What is an evaluation? Why does UN-Habitat carry out evaluations? What is evaluation in the UN system and what are UN-Habitat’s expectations, roles and responsibilities in evaluation?
18 The Role of Evaluation in results-based management ( UNEG/REF(2007)1 19 Secretary-General’s Bulletin, “ Regulations and Rules Governing Programme Planning, the Programming Aspects of the Budget, the Monitoring of Implementation and the Methods of Evaluation (PPBME)”, ST/SGB/2000/8 20 UN-Habitat Evaluation Policy was approved by UN-Habitat Management in January 2013. It is available at http:// www.unevaluation.org/evaluations 21United Nations Evaluation Group (UNEG), “Norms and Standards for Evaluation in the UN System”, April 2005 (available online at http://www.uneval.org). 22“Revised UN-Habitat Evaluation Framework”, approved by UN-Habitat Management in September 2015 and adopted by Executive Directive of 5th January 2016.
4.1.1 What is evaluation
UNEG defines an Evaluation as an assessment, as systematic and impartial as possible, of an activity, project, programme, strategy, policy, topic, theme, sector, operational area, institutional performance, etc.
It focuses on expected and achieved accomplishments, examining the results chain, processes, contextual factors and causality, in order to understand achievements or the lack thereof. It aims at determining the relevance, impact, effectiveness, efficiency, sustainability and impact of the interventions and contributions of the organizations of the UN system.
An evaluation should provide evidence-based information that is credible, reliable and useful, enabling the timely incorporation of findings, recommendations and lessons into the decision-making process of the organizations of the UN System and its Members.
4.1.2 Why UN-Habitat carries out evaluations
Evaluations in UN-Habitat are carried out to inform the management, the governing bodies and the agency’s partners about what UN-Habitat is achieving, what improvements should be considered, and what is being learned. In general, the main purposes of the evaluation will relate to programme improvement, knowledge generation, learning and accountability.
- Programme improvement: Evaluation is an important source of evidence of achievement of results and organizational performance.
- Decision-making: The evidence can be used for decision-making on programme direction, allocation of resources and improving programmes.
- Knowledge generation and supporting learning: Evaluation is an important contributor to building knowledge and to organizational learning. Some evaluations are undertaken to describe the effects of an intervention, and as such contribute to building knowledge and organizational learning. This may form a basis for making future interventions more relevant and effective.
- Supporting accountability and transparency: Sharing evaluation results with key audiences demonstrates accountability and transparency. By building a greater understanding about what UN-Habitat is intending to achieve, evaluation meets requirements for donors, governing bodies and Member States and generates support.
- Agent of change: Evaluation promotes, defends or opposes specific approaches or programmes and shapes opinions.
- Cohesion and collaboration: Evaluation informs the planning, programming, budgeting, implementation and reporting cycle, and increases consistency and communication between departments and organizations.
- Evaluation aims at improving institutional relevance, optimizing the use of resources, providing client satisfaction and maximizing the impact of the contribution of the organization.
In essence, evaluations offer a learning opportunity to find out what is working, what is not and what needs to be improved. Evaluations demostrate objectivity in identifying valid, balanced and accurate results that are supported by the evidence assessed.
4.1.3 Evaluation in the UN System
Within the specific context of the UN, evaluation helps to ensure the accountability of the various UN bodies, their managers and staff, to the General Assembly (GA) and/or to their respective governing bodies, as well as to national stakeholders. At the same time, it supports reflection and learning by Member States, governing bodies, management and staff, as well as national stakeholders, on the relevance, effectiveness, efficiency, impact and sustainability of UN activities, in order to improve on them.
Evaluation serves this dual purpose of accountability and understanding what we are doing right and what we may be getting wrong through the provision of reliable and credible evaluative evidence, analysis and information to Member States, the Secretary-General, programme managers, staff, and national stakeholders, on the activities of the UN system and their impact. These evaluation outputs are provided in the form of evaluation reports, briefings, various information exchanges and other evaluation products.
Because evaluation has to simultaneously support both accountability and learning at different levels of governance, oversight, management and operations, it has to be conducted at different levels within each organization. In many organizations, evaluations are carried out as either centralized or decentralized functions.
The regulations that currently govern the evaluation of United Nations activities were promulgated on 19 April 2000 in the Secretary General’s bulletin (PPBME) and updated in 2016 as ST/SGB/2016/6. In 2005, the heads of evaluation of 43 UN entities, under the auspices of the UN Evaluation Group (UNEG), adopted a common set of norms and standards for evaluation in the UN system.
With regard to evaluation of UN Secretariat programmes, the rules and regulations of the PPBME apply for all of them. Given the heterogeneity and size of Secretariat programme activities, and in order to ensure that all programme activities are evaluated, evaluation is decentralized to the programme level, and each Secretariat programme is required to conduct regular, periodic evaluation of all activities.
The central evaluation function of the Secretariat is assigned to OIOS, where evaluation complements its other oversight functions of investigation, audit and inspections, by focusing on broad issues of relevance, effectiveness, efficiency and impact of Secretariat programmes and activities. The Evaluation Section of OIOS is mandated by the GA to conduct in-depth and thematic evaluations of the work of the Secretariat programmes, as well as to establish guidelines for the conduct of self-evaluation by the programmes, and to provide methodological support.
In the General Assembly report “Strengthening the role of evaluation findings in programme design, delivery and policy directives” (A/59/79), OIOS found that programmes did not consistently use the same nomenclature when classifying evaluations. In June 2005, OIOS published an online manual “Managing for results: A guide to using evaluation in the United Nations Secretariat”, which defines four types of evaluations. These are:
(a) Mandatory external evaluations, which are requested by the General Assembly, Committee for Programme and Coordination (CPC) and donors; conducted by OIOS, the Joint Inspection Unit (JIU) and external consultants; and used by the CPC, General Assembly, and other intergovernmental bodies.
(b) Discretionary external evaluations, which are requested by senior managers, conducted by OIOS, JIU or external consultants, and used by donors, external stakeholders and senior managers.
(c) Discretionary self-evaluations, which are requested by senior managers and subprogramme managers, conducted by staff internal to the departments, and used by senior managers and their staff.
(d) Mandatory self-assessment, are required by management and conducted by project managers in the context of the focused approach to reporting results of sub-programme performance and integrating lessons into management decisions.
The above are the different types of evaluations conducted by UN-Habitat, which can further be differentiated based on:
(a) When the evaluation is undertaken (timing – before, during and after): (i) ex-ante evaluation, (ii) mid-term evaluation, (iii) terminal evaluation, and (iv) impact/ex-post evaluation;
(b) What is being evaluated: (i) project evaluation, (ii) programme evaluation, (iii) thematic evaluation, (iv) policy evaluation, (v) sectoral evaluation, (vi) institutional evaluation, (vii) country evaluation;
(c) Who conducts the evaluation: (i) self-evaluation, (ii) independent evaluation, (iii) joint evaluation, (iv) peer review; and
(d) How the evaluation is conducted: (i) Summative and (ii) Formative evaluations
Self Evaluations
Self-evaluation is a requirement for all closing projects, in line with the implementation of the UN-Habitat Evaluation Policy and the Revised UN-Habitat Evaluation Framework, expected to further strengthen the role of evaluation in project/programme design, implementation and decision-making.
The main purpose of institutionalizing self-evaluations is to increase the coverage and scope of UN-Habitat evaluations, within the budget constraints affecting the organization. Self-evaluations facilitate learning from projects and use of the evaluation findings. A results-based Self-Evaluation (RBSE) template (annex 14) should be used by project managers to assess performance of the projects they supervise.
4.1.4 UN-Habitat’s expectations from evaluations
UN-Habitat expects evaluations to bring improvements to its interventions through better informed decision-making. Evaluations should provide information about which expected accomplishments of the evaluated intervention are being achieved, what improvements should be considered and what is being learned.
Evaluations should be used to determine what works and what does not work – in achieving sustainable development. Producing credible, timely and useful evaluations that describe how UN-Habitat’s interventions are performing promotes effective programming. What is learned from evaluations becomes strategic in decision-making, and in turn leads to improved policies, strategies and operations.
Evaluations satisfy the requirements of the Terms of Reference and expectations set out in the evaluation work plan. They bring focus to UN-Habitat’s mandates, programming priorities and cross-cutting themes (e.g., gender equality, youth, human rights and the environment). They should produce credible, reliable results using an appropriate design and adhering to appropriate methods and techniques.
4.1.5 Roles and responsibilities
The UN-Habitat Evaluation policy outlines the organizational roles and responsibilities of key constituents of UN-Habitat (see pages 11-13 of the UN-Habitat Evaluation Policy). Table 23 summarizes the roles and responsibilities that are relevant to all evaluation processes:
The evaluation manager: The primary role of the evaluation manager is to manage the evaluation process, rather than conduct the evaluation. Typical tasks of the evaluation manager include preparing the terms of reference, establishing the evaluation team, overseeing the review of draft reports, disseminating evaluation results and making other logistical arrangements. The evaluation manager is responsible for representing the best interests of the agency and is accountable for his/her decisions.
In UN-Habitat, management of centralized evaluations is the responsibility of the Chief of the Evaluation Unit. For decentralized evaluations the responsibility is that of programme/project managers, and supervision falls under the relevant branch coordinator or head of office. The Evaluation Unit provides guidance and support to the evaluation process.
The evaluator or evaluation team: conducts the evaluation through document reviews, interviews, surveys, meetings, site visits, etc. The team is generally comprised of one or more external consultants. The candidates should have the knowledge and working experience to carry out the evaluation as per Management’s expectations. The evaluation team leader should fully understand UN-Habitat’s work and have the capacity to effectively address the four cross-cutting issues (i.e., gender equality, human rights, climate change and youth) in the evaluation methods, findings and recommendations.
Reference Group: Resources permitting, the evaluation should have a reference group comprised of key stakeholders who work closely with the evaluation manager to guide the evaluation process and ensure the quality of the process and outputs of the evaluation. The reference group should review the documents required such as draft TOR, draft evaluation work plan and draft evaluation reports and provide advice on quality and options for improvement.
UN-Habitat Management Board/Senior Management Team: A management response, as the formal, written response to the findings and recommendations of an evaluation, is discussed and adopted by UN-Habitat’s management. The management response is formulated jointly by organizational entities that are responsible for, or will be involved in the follow-up to the evaluation recommendations. The management response is a requirement for all centralized evaluations managed by the Evaluation Unit.
4.1.6 Involving key stakeholders in evaluation processes
Involving stakeholders before an evaluation starts, and keeping them informed about its progress during the evaluation process allows the stakeholders to explain their expectations of the evaluation and raise related questions and concerns. This involvement is crucial to ensuring the support of stakeholders during the evaluation process and later during the implementation of follow-up actions to the evaluation.
Stakeholders in an evaluation should be identified in the evaluation TOR, and should ideally be involved in the preparation of the TOR. One mechanism for ensuring the active involvement of stakeholders in an evaluation process is through the establishment of a reference group. The reference group can be formed in order to provide the evaluator or evaluation team with feedback from a technical and methodological perspective. Reference group members can include stakeholders and peers, both internal and external to the project and to UN-Habitat. The composition of the reference group is at the discretion of the evaluation manager.
The reference group performs an oversight function that helps ensure transparency of the evaluation process, as well as generate a sense of ownership and participation among reference group members and the organization as a whole. Box 24 explains why stakeholders should be engaged.
