The implementation of evaluations is carried out by the evaluation team. The evaluation manager stays in touch with the evaluation team to provide assistance or clarification where needed. To ensure that evaluations are carried out in a professional and ethical manner28, evaluators should be provided with UNEG Norms and Standards as an integral framework for evaluations in UN-Habitat, in addition to UN-Habitat policy.

After the evaluation workplan is approved, establishing a clear understanding of how the evaluation will be carried out and what will be achieved, the evaluator embarks on data collection and analysis.

28 UNEG Ethical Guidelines and UNEG Code of Conduct for Evaluations in the UN System (annex 10 and 11).

4.3.1 Conducting the evaluation

(a) Data collection

The evaluation team conducts the evaluation following the methodology described and agreed upon in the TOR and inception report.

The data to be collected and methods for collecting the data will be determined by the evidence needed to address the evaluation questions, the analyses that will be used to translate the data into meaningful findings and judgments about what data is feasible to collect given time and resource constraints.

UN-Habitat evaluations should draw heavily on performance data generated through monitoring during the programme or project implementation cycle. However, performance information and indicators do not explain the full range of questions the evaluation seeks to address. For example, the indicators provide a measure of what progress has been made. They do not explain why that progress was made or what factors contributed to the progress. UN-Habitat evaluations should make use of a mix of data sources, collected using multiple methods, to give meaning to what the performance information reveals about the intervention.

Primary data consists of information evaluators observe or collect directly from stakeholders about their first-hand experience with the intervention. This data generally consists of the reported or observed values, briefs, attitudes, opinions, motivations and knowledge of stakeholders, generally obtained through questionnaires, surveys, interviews, focus groups, key informants, expert panels, direct observation and case studies. These methods allow for more in-depth exploration and yield information that can facilitate deeper understanding of observed changes in outcomes and outputs.

Secondary data is data that was collected, compiled and published by someone else other than the stakeholder. Secondary data can take many forms but usually consists of documentary evidence that is directly relevant to the evaluation. Sources of documentary evidence include: national demographic data, published reports, project or programme plans, monitoring reports, previous evaluations/reviews and other records.

Table 29 presents brief descriptions of data collection methods that are commonly applied in evaluations in UN-Habitat.

Data analysis and synthesis

Data analysis is a systematic process that involves organizing and classifying the information collected, tabulating and summarizing it, and comparing the results with other appropriate information to extract useful information that responds to the evaluation questions and fulfils the purpose of the evaluation.

Data analysis seeks to detect patterns in evidence, either by isolating important findings (analysis) or by combining sources of information to reach a greater understanding (synthesis). Mixed-method evaluations require the separate analysis of each element of evidence and a synthesis of all sources in order to examine patterns of convergence or complexity.

Depending on the evaluation, the evaluation team discusses the main findings with the evaluation manager or presents the main findings to the relevant UN-Habitat staff members. It is important that the evaluation manager ensures the independence of the evaluators by being prepared to accept the findings, even when they differ from the programme or evaluation manager’s perspective.

4.3.2 Preparation of the draft evaluation report

Virtually all evaluations are presented as written reports. The main objective of the evaluation report is to convey the results of the evaluation in a way that corresponds to the information needs of the intended users of the evaluation.

Throughout the evaluation process, the evaluation team will document findings and conclusions. Usually, the lead evaluator will organize and facilitate team meetings to discuss findings and conclusions and coordinate the preparation of a draft report.

(a) The format of the draft report

UN-Habitat recommends the format for evaluation reports. The format should be used unless there is good reason for doing otherwise. The evaluators should consult with UN-Habitat’s evaluation manager before adopting a different framework. Table 30 summarizes the format and contents of the evaluation report.

(b) Review of the draft evaluation report

The evaluation manager sends the draft report to the relevant branches and office managers and to other programme or project staff for comments. Depending on the evaluation, the draft report may also be sent to external stakeholders for comment.

Comments can focus on the conclusions and recommendations, as well as technical and methodological issues. It is the responsibility of the relevant programme or project officers to conduct a technical review with inputs from other stakeholders, which includes:

• Is the information in the report accurate? (i.e., check for factual errors);
• Is the information in the report complete? (i.e., is there information lacking that could affect the conclusion);
• Are the recommendations relevant, objective and specific enough to be implemented?

For all evaluations, the evaluation manager conducts a methodological review or quality check of the draft report. This review aims to ensure that the report and the drafting process meet a set of standard quality criteria (see table 31 below).

The evaluation manager sends the compiled comments to the evaluation team for incorporation.

UN-Habitat Quality Checklist for Evaluation Reports

This checklist is intended to help evaluation managers and evaluators to ensure that the final product of the evaluation (evaluation report) meets the expected quality.

(c) Preparation of the final evaluation report

The evaluation team adjusts the report based on feedback provided and submits the final report to the evaluation manager. The evaluation manager ensures that the report is edited (in most cases only the executive summary is formally edited) and formatted properly. In case of major edits, the evaluators should review the report once more to ensure that they have not affected the content. Next, evaluators sign off on the report and no further changes may be made to the report.

Clearance and approval of the Evaluation Report

The evaluator submits a soft draft evaluation report to the evaluation manager for review and clearance.

Once UN-Habitat’s needs have been addressed satisfactorily, the UN-Habitat Management Board approves the final report and the formulation of UN-Habitat’s management response and a follow-up action plan to its findings, conclusions and recommendations is developed.