Prior agreement on the purpose of the evaluation;
The basic questions of ‘what, where, when and why’ can be answered;
It is undertaken by a capable team, able to meaningfully seek the views of target groups as well as to interpret statistical data;
The results and recommendations are presented and discussed in such a way that they are likely to be used.
Effective monitoring, evaluation and learning systems in nutrition are of paramount importance as different sectors may be contributing to nutrition outcomes. Each sector must be able to monitor its own contribution to the changes being made. In particular, there is no clear evidence as to which food security actions are most likely to result in nutritional benefits, so there is need for strong monitoring and evaluation systems to identify this learning.
There are also several challenges to effective monitoring and evaluation. One key challenge is to attribute change to a specific programme (i.e. the programme caused the change). Where comparisons are made to non-intervention (or control) groups, then the selection of these control groups is critical to ensure that direct comparisons can be made legitimately. Another important challenge is the existence and quality of baseline data to assess progress in meeting project objectives. Most large-scale nutrition surveys carried out in developing countries have been conducted as part of national or regional exercises that are independent of projects. At the same time, a growing number of large-scale projects are developing their own, non-standardised, monitoring and evaluation systems that include periodic surveys to assess whether project objectives are being met. In both cases, the use of such ‘baselines’ to attribute changes to a project or programme can be problematic and tenuous.
Effective monitoring and evaluation is of particular concern in emergencies, where there is, typically: a lack of standardisation of methodologies and indicators; no agency with a mandate to act on the findings; and limited time for establishing baseline information.73 However there are guidelines for monitoring and evaluating nutrition interventions (see Annex 7).
Sources of Further Information
Evaluation methods for the European Union’s external assistance. EuropeAid Co-operation Office, (2006) http://ec.europa.eu/europeaid/evaluation/methodology/index_en.htm
Methodological bases for evaluation, Volume 1.
Guidelines for geographic and thematic evaluations, Volume 2.
Guidelines for project and programme evaluation, Volume 3
Impact Assessment of Large Scale Food Security Programmes. E-learning course by FAO in collaboration with Wageningen University and Research Centre for Development Innovation (2010)
Impact Evaluation in Practice. Paul J. Gertler [et al.] The World Bank, 2011. Available as an interactive textbook at /pdt
Monitoring and Evaluation. A Guidebook for Nutrition Project Managers in Developing Countries. Human Development Network, the World Bank, Sept 1999
OECD, Development Assistance Committee (DAC), Evaluating Development Co-operation. Summary of Key Norms andstandards
Monitoring and evaluation of nutrition and nutrition-related programmes. A training manual for programme managers and implementers. The Applied Nutrition Programme, University of Nairobi School of Nutrition Science and Policy, Tufts University, Aug 2000
Sphere Project, 2011: The Humanitarian Charter and Minimum Standards in Disaster Response /
A guide to monitoring and evaluating policy influence. ODI Background Notes, February 2011
CHAPTER 4: NUTRITION IN AID DELIVERY METHODS
Source: Muriel Morisson, GRET
Having explored how nutrition can feature in specific thematic areas, national and donor strategies, this chapter reviews the implications of using different aid delivery methods. The choice of method needs to be considered in light of context-specific institutional funding opportunities. The ultimate objective remains to support the Government to develop and implement a nutrition-sensitive national policy/strategy.
Figure 7 presents the three approaches that tend to be used and their related financing modalities.
Figure 7: Aid delivery methods used by the European Commission
Source: Adapted from Guidelines on the Programming, Design and Management of General Budget Support, EC, revised version 2009, unpublished.
Project Approach EC procurement and grant
Sector Approach Common pool funds
General/Global Approach Budget Support
In terms of sector approaches, there are several different models on how funds can be managed. Box 14 provides a summary of those most commonly used.
Box 14: Sector Terminology
A programme-based approach (PBA)is a way of engaging in development cooperation based on the principle of coordinated support for a locally owned programme of development. This could be a national poverty reduction strategy, a sector programme, a thematic programme or a programme of a specific organisation.
A sector-wide approach (SWAp) is usually seen as a programme-based approach operating at sectoral level. It defines all significant funding that supports a single sector policy and expenditure programme.
A sector programme is a government-owned programme, based on a sector policy and strategy, a sector budget and a sector coordination framework. The Commission’s aid instrument for supporting a sector programme is known as a Sector Policy Support Programme (SPSP). PBA, SWAp and SPSP could be implemented through either project, sector budget support or general budget support.
A pool fund receives contributions from different external agencies, and in certain cases from Governments, to finance a set of eligible budget lines or actions to support a sector programme.ach aid delivery method offers a fresh opportunity to introduce and embed nutrition-related concerns and factors. The process of working through each method tends to include several key steps that are common to all methods. These are summarised in Figure 8.
Figure 8: Making aid delivery methods nutrition-sensitive
This process provides a backdrop for exploring specific steps and issues relevant to different aid delivery methods. Three approaches are covered in the two sections that follow. The first (4.1) covers general/global and sector approaches. These two approaches are discussed together since they are both financed by budget support. The second section (4.2) covers the project approach - which is further subdivided into development and humanitarian projects.
4.1 Guidance for addressing nutrition through general and sector approaches
The national strategies of partner countries are usually focused on poverty reduction. But successful poverty reduction – and ultimate alleviation – is likely to require specific focus on nutrition (see 1.2.2 on the impact of undernutrition on the national economy). This, therefore, needs to be recognised in the early dialogue with government to inform the decision about which aid approach to use. Once a general or sector approach is selected, some of the issues considered during programming stage will have to be reviewed and developed with the government and other stakeholders.
These discussions should explore the best way to incorporate nutrition in poverty-reduction efforts:
Does the government recognise the challenge of undernutrition? This is the most important hurdle, for without government buy-in, nutrition risks being an insignificant add-on. Sensitisation to the importance of nutrition is essential to securing meaningful government engagement. Evidence of the scale of undernutrition in the country, perhaps tied to econometric models of its impact at national level, are extremely powerful aids. But such analysis is rare, in which case some of the statistics and models presented in Chapter 1 could be used.
Is there a strong policy framework, with associated budget allocation? If not, could it be incorporated into the cooperation agreement?
Are programmes with nutrition objectives/outcomes in place or planned? If not even planned, these could be phased in through the preceding steps to develop a policy framework.
Are nutrition-related indicators included in the Performance Assessment Framework?
Whether through a general or sector approach, multiple institutions are likely to be required to effectively tackle the multi-faceted nature of undernutrition. In this light, the following guidance may be of help:
Solid mechanisms for sector and donor coordination are critically important. Many actors are likely to be involved, (both within and outside government, operating at national and sub-national levels), so an institutionalised national nutrition coordinating body or council could be needed.
The institutional mandate for overall coordination has to be well thought through. A central ministry (e.g. Finance or Planning), a minister with more general responsibility or a high-level office outside any ministry (as discussed earlier) may play that role.
Similarly, good governance is heavily dependent on effective action at sub-national levels, in support of decentralisation initiatives.
Tables 4a and 4b summarise the processes and incremental steps to incorporating nutrition in general/global and sector approaches. Guiding questions provide prompts on how to do this at each phase. Even before phase 1, however, an important preliminary step is to integrate nutrition objectives and indicators in the country’s development strategy/national framework, as discussed in Chapter 3.
Table 4a: General/Global approach: Steps to incorporating nutrition
Actions to be taken
1. Analysis and scoping (Identification)
Government eligibility for budget support
Understanding government’s position on nutrition
Consistency with EU policies/strategies and the Aid Effectiveness agenda
Issues and state of play
Risks and assumptions
Next steps, work plan, and time schedule
Has a nutrition situation analysis been undertaken? If not, why not and could one be planned?
How do national or sector policies/strategies refer to nutrition? Is there scope for strengthening them?
How are national or sector policies/strategies likely to have an impact on maternal and child nutrition?
Are nutrition indicators available in the national policy and strategy?
Are nutrition indicators included in PAF (Performance Assessment Framework)?
Are there donor allies who are concerned about nutrition?
Is the government positioned to coordinate nutrition across different sectors?
Are there specific capacity gaps (skills and/or processes such as nutrition monitoring systems) to be addressed?
2. Design (Formulation)
Rationale, Country description
How is nutrition featured in the contextual analysis?
Which nutrition indicators could be linked to disbursement?
Are these indicators measured annually?
Does the government oversee these indicators?
Are chosen indicators coherent with the country context?
Have existing analyses of undernutrition and food insecurity been included as supporting documents?
Is there a clearly developed framework for sourcing nutrition-related information and for verifying its reliability?
3. Implementation and monitoring
Matrix of performance indicators
Coherence with the MDGs
Ensure the quality of performance indicators
Are any of the MDGs most relevant to nutrition (MDGs 1, 2, 4 and 5) a priority for the government?
Is the nutrition target for MDG1 used by the government (i.e. halving the proportion of underweight children aged 5 years or below)? Could it be?
Evaluation should be government led and include other donors providing budget support
The conceptual framework for undernutrition offers useful guidance on what to assess.
Table 4b: Sector approach: Steps to incorporating nutrition
Actions to be taken
1. Analysis and scoping (Identification)
Assessment of the status of the sector approach and the consensus and readiness to develop a sector programme.
Preliminary assessment of the quality of the sector programme through the 7 areas of assessment:
How could this sector help yield nutrition benefits?
What would the implications be for coordination and monitoring?
Does the sector have a well-formulated policy with nutrition implications? Does it link to the national poverty analysis?
Is nutrition an objective of the sector or an outcome indicator? If not, could it be? Would other donors support it?
Would nutrition concerns influence any targeting decisions? (e.g. in prioritising support to areas with higher levels of undernutrition)?
To what extent could nutrition concerns be addressed through a multi-stakeholder working group?
Capacity building, strength of the sector (in terms of budget availability and dedicated personnel)?
Does the work of other development partners include nutrition objectives?
What proportion of the sector budget would be aligned to nutrition outcomes?
Are nutrition indicators included in the PAF?
What cooperation and coordination is required across different sectors (e.g. to use information from other sector sources, to seek technical support)?
Preparing a financing proposal
3. Implementation and monitoring
Include sector policy support as a contribution to sector programme.
Could nutrition indicators relevant to the chosen sector(s) also be compatible with the poverty alleviation strategy?
Assess relevance, efficiency, effectiveness, impact and sustainability of programme, its added value in helping achieve the sector goals and the appropriateness of chosen implementation modalities.
Has the policy support resulted in nutrition-related outcomes (as per objective and/or indicators agreed)?
Indicators appropriate for general and sector approaches
For both general and sector approaches, careful consideration will need to be given as to which nutrition-related indicators to include. This will be guided by discussions with government and other stakeholders, the information context, national capacity (for data gathering and analysis) and budgets. Annex 3 sets out the nutrition-specific indicators that have been used internationally, with guidance on their interpretation and suitability for the different aid delivery methods. Box 15 draws from the list in Annex 3 and lists the indicators considered to be most relevant to General and Sector approaches. Only indicators with specific added value to nutrition are listed, and are intended to complement other indicators that might be included in general and sector approaches. Indicators should be drawn from national policies and strategies; they should not be imposed. It is hoped that the annex provides ideas and stimulates careful consideration of what might be appropriate in a particular context.
Box 15: Key nutrition indicators for general and sector approaches
The performance targets for each indicator would normally be established on an annual basis. When selecting the indicators and their targets, attention should be paid to avoiding potential problems, such as the absence of precise and unambiguous definitions, lack of coherence between the calendar of the Financing Agreement (for assessment of performance) and that of national processes (including availability of data) which may result in delaying disbursements or lack of comparability of the indicators from one year to the next. See case study on Mauritania – the challenge of integrating nutrition indicators in budget support, in separate file.
The sources for each of the indicators should be clearly identified and, as much as possible, draw on data produced by the national statistical system (avoiding ad-hoc or project-related indicators). The methodology used to calculate each indicator should be clearly described, including that of aggregated data, so that indicators are reliably comparable from one year to the next. The sensitivity of each indicator to policy changes will also need to be assessed, so that the time-schedule of monitoring corresponds to the likely time necessary to see desired improvements.
Choosing targets for the indicators
A clear baseline will be required, against which progress can later be evaluated.
The targets should be drawn from national and/or sectoral strategies, and should be coherent with international objectives (especially the MDGs 1, 4 & 5);
The composite governance indicators (see Annex 3) are relevant to objectives concerning the development and implementation of national nutrition policies and strategies. The precise mix of indicators from all the possible options will depend on the context.
Peru:Sector budget support for the Peruvian Nutritional Programme
National development policy
In 2002, under the name of Acuerdo Nacional (National Agreement), the government together with the main political parties and civil society organisations agreed to a new approach where State policies would be developed through a consensual model. All relevant stakeholders would agree a shared vision for the policy and actively participate in the policy development process. The Acuerdo Nacional covers policies on poverty reduction and on food security, it prioritises support to vulnerable populations, socially excluded groups and people living in extreme poverty and is based on a holistic concept of human development.
An integrated strategy to fight poverty – Crecer (meaning ‘to grow’) – was shaped on the principle that only a common approach, which includes all relevant actors and different types of intervention, could effectively reduce poverty. Crecer prioritises the poorest rural areas with high child undernutrition rates to receive budgetary distributions. The key element of this strategy consists of multi-sector interventions which combine centralised governmental execution with decentralised implementation at local/regional levels. The Crecer strategy comprises several programmes tackling poverty, including mother and child health, nutrition, basic education and identity documents.
The Peruvian Nutritional Programme (PAN) is one of the programmes developed under Crecer. Undernutrition is considered to be a crucial cause of poverty and a social cohesion gap. The PAN’s goal is to reduce undernutrition from 25 % (in 2005) to 16 % (in 2011) and the 2009 budget amounted to €269 million. The logical structure of PAN, following a product-result-impact chain, allows follow up and measurement of indicators. The Commission committed €60.8 million at the end of 2009 for the implementation of PAN, through sector budget support (economic and finance), focusing on the three poorest regions of Peru. In early 2010, in order to increase the responsibility of these regional governments, the Ministry of Economy and Finance signed with each one a Sector Budget Support, adapted to local public finance management. This gave them responsibility to implement health policy at regional level. This initiative aims to consolidate the decentralisation process and increase incentives to achieve the objective of reducing undernutrition.
Performance monitoring and criteria for disbursement
The disbursements of fixed instalments is conditional upon a positive evaluation of the macroeconomic situation, the satisfactory implementation of a PFM (Public Finance Management)-improvement action plan and satisfactory implementation of PAN. Other specific conditions included are: i) improved public access to and transparency of information regarding strategic programmes; and ii) setting of annual targets for the indicators of variable instalments. Variable instalments will be measured through indicators previously selected in agreement with relevant stakeholders. For instance, the percentage of children under 24 months of age enrolled in the integral health insurance with dietary iron supplement will be measured. The proportion is expected to increase from 4.5 % (2009 baseline) to 59.5 % in 2013.
For further reading see: http://ec.europa.eu/delegations/peru/eu_peru/tech_financial_cooperation/index_en.htm
Source: Case study prepared by the authors
EC Guidelines on the Programming, Design & Management of General Budget Support, 2009 http://ec.europa.eu/europeaid/how/delivering-aid/budget-support/index_en.htm
DG ECHO Interim Position on Nutrition in Emergencies, 2010 (internal document)
Communication: Humanitarian Food Assistance, 2010 http://ec.europa.eu/echo/files/policies/sectoral/Food_Assistance_Comm.pdf
EU Communication: The EU Role in Global Health, 2010 http://ec.europa.eu/development/icenter/repository/COMM_PDF_COM_2010_0128_EN.PDF
EU Communication: An EU policy framework to assist developing countries in addressing food security challenges, 2010 http://ec.europa.eu/development/icenter/repository/COMM_PDF_COM_2010_0127_EN.PDF
Tools and Methods Series: Guidelines No 2. Support to Sector Programmes Covering the three financing modalities: Sector Budget Support, Pool Funding and Commission project procedures. EuropeAid, July 2007. http://ec.europa.eu/development/icenter/repository/Support-to--Sector-Programmes_short_27072007_en.pdf
Commission concept note: ‘ Social Transfers: an effective approach to fight food insecurity and extreme poverty’, 2010 http://ec.europa.eu/europeaid/infopoint/publications/europeaid/186a_socialtransfer_en.htm
Commission concept note: Enhancing EC’s contribution to address Maternal and Child undernutrition and its causes, 2009 http://ec.europa.eu/europeaid/infopoint/publications/europeaid/137a_en.htm
4.2 Guidance for addressing nutrition through projects
A project is a series of activities that aim to attain clearly specified objectives within a defined time-period and budget.
4.2.1 Guidance for addressing nutrition through development projects
Strategies and actions to improve nutrition need to be developed according to specific country needs, resources, circumstances and the development project objectives.
The table below summarises the steps in developing project support. Questions have been inserted for each phase to stimulate ideas on how nutrition can be incorporated.
Actions to be taken
1. Analysis and scoping (Identification)
Assessment of the nutrition context
Scrutiny of proposals
Agree with the Government and relevant stakeholders that proposed actions are appropriate
Assess partner’s capacity and own resources
Preliminary assessment of the most appropriate financing modality
Prepare and commission an assessment mission
Initiate internal quality control mechanism, e.g. quality assurance at country level.
Is nutrition a priority concern? Does it need to be?
Who is worst affected by undernutrition? Where? What are the likely causes (c/f conceptual framework).
Trend: How has the nutrition situation changed over time?
Proven skills/experience of partners in nutrition?
Is there any nutrition coordination at Government level and amongst stakeholders? Is there a shared analysis of the problem?
Is there agreement on the need to respond in nutrition?
2. Design (Formulation)
Detailed project description (situation analysis, project description, management arrangements, feasibility and sustainability)
Prepare and conclude financing agreement
Are nutrition objective/outcomes integrated in the project design and log-frame?
What actions need to be taken to ensure links with others sectors relevant to nutrition?
3. Contract with implementing partners
Description of project including specific deliverables and monitoring/reporting requirements. Prepare and conclude financing agreement with the Government, international organisation or civil society
Is there coherence between the nutrition objectives/outcomes and indicators defined in the financial agreement, the nutrition objectives/outcomes/impact and indicators defined by the implementing partner?
4. Implementation and Monitoring
Provide timely finance, management and technical support to monitor project implementation and ensure an appropriate level of accountability for resources used and results achieved, and to identify and learn lessons from implementation.
Are nutrition indicators agreed and appropriate?
How will data on these indicators be derived? Geographic coverage? Timeliness?
Assess with government and partners the relevance, efficiency, effectiveness, impact and sustainability of the programme, the appropriateness of chosen implementation modalities
Ensure that evaluation conclusions and transferable lessons are acted upon and fed back into future policy making and programming.
Has the project resulted in nutrition-related outcome/impact (in line with the objective and/or indicators)?