--Greg Armstrong --
Problems with United Nations agency reporting on results can, in part, be attributed to ambiguous definitions of Outputs.
Level of Difficulty: Moderate-complex
Primarily useful for: Anyone trying to understand the UN’s inconsistent RBM system
Coverage: 13 papers, totalling 721 p.
Most useful components: Technical Briefs on Outcomes, Outputs, Indicators and Assumptions
Limitations: Large number of potential documents laden with bureaucratic language
Who this is for
This post, and the previous review of UN agency problems in reporting on results, is intended for bilateral aid agency representatives, national government representatives, project managers, monitors or evaluators, trying to understand the inconsistent reporting of project results by UN agencies. Because the UN documents are often lengthy, and laden with bureaucratic language, it is unlikely to be of interest to people who don’t need to work with UN agency counterparts.
Background: Problems in UN RBM
This is the second of four posts assessing UN agency results chains, results definitions and problems in reporting on results, and (in the third and fourth posts) the results frameworks bilateral aid agencies use. In the previous post, I suggested that inconsistent UN agency results reporting could, in part, be attributable to a weak results culture, and sometimes weak leadership at the country level within some UN agencies. This post reviews how ambiguous results definitions also undermine UN agencies’ credibility in results reporting.
A third post in this series will review results chains for 3 bilateral aid agencies – SIDA, AusAID, and DANIDA -- define results, and the fourth and final post will review how USAID, DFID, CIDA and EuropeAid define results and results chains.
Language matters. I have argued elsewhere that the terminology used in Results-Based Management is dysfunctional, largely because the jargon (Outputs, Outcomes, Impact, Objectives, Purpose) can mean many different things and, that in the context of development programming, terms used for results are intended to mean something different than they mean in day to day usage. What works in almost all project contexts, however, is to focus on change as the characteristic of a result. This is a word and a concept that works in most languages, and appeals to people’s desire for common-sense terminology.
For most bilateral donors, whatever the specific terms they use, completed activities -- often referred to as Outputs -- are not sufficient for reporting purposes. While bilateral project managers are obviously required to report on completion of activities, the real emphasis in project reports is expected to be on if and how these activities are contributing to significant changes in the short to mid-term -- changes to knowledge, attitudes, policy or professional practice.
In other words, results.
Some agencies refer to these results as Outputs, Outcomes and Impacts. Others refer to them as Objectives, or Purpose, or as Immediate, Intermediate and Ultimate Outcomes. But whatever the terms, the focus is clear: “Tell us about how the project or programme is contributing to change, not just about how you spent the money”.
The problem for bilateral and national agency partners trying to hold UN partners to reasonable standards of results based management lie, I think, in the vast number of documents dealing with results in the UN context; the ambiguity of the UN definitions of results; and the confusion about how results chains for projects relate to results chains at the country programme level for different agencies.
At the UN Development Assistance Framework level, results from different agencies are essentially being aggregated, and as the January 2010 UNDAF document "Standard Operational Format and Guidance for Reporting Progress on the UNDAF" [22 p.] made clear, the UNDAF report should be “focused on reporting results at a strategic level….”. [Update: That UNDAF guide no longer appears easily accessible on the UNDAF website, but was replaced apparently, in 2011 by a longer UNDG Results-Based Management Handbook.]
Unfortunately, terms that define results for aggregations of projects at the strategic level do not necessarily work at the individual project level.
So, how did the UN, in the UNDAF and UNDG guides and technical briefs, deal with results?
Results chains describe the sequence, and the nature of links between, activities, completed activities, and near-term, mid-term and long-term results. Most practitioners agree that a direct cause and effect between activities and results is unreasonable given the wide range of intervening variables that occur in real life, but that results chains describe the general sequence of how activities can contribute to change – or results.
The several UN documents reviewed here and in the previous post variously refer to results chains (sometimes in the same document) as
Activities→Outputs→Outcomes→Impacts
Activities→Outputs→Agency Outcomes→UNDAF Outcomes→National Priority
Activities→Outputs→Country Programme Outcomes→UNDAF Outcomes→National Priority
Activities→Outputs→Agency Outcomes→Country Programme Outcomes→UNDAF Outcomes→National Priorities
“Activity Results”→Outputs→Outcome→UNDAF Outcome→National Priority
UN Agency Outputs→UNDAF Output→National Outcome→National Goal
Looking at these it is no wonder that there are differences among implementing agencies in how results are explained for projects and programmes in the UN system.
Most UN agencies use as the basis for their own definitions of results, the 2003 harmonized UNDG Results-Based Management Terminology [3 p.] which grew out of the OECD/DAC Glossary of Key Terms in Evaluation and Results-Based Management [37 p.]. “Outcomes,” the harmonized terminology states “represent changes in development conditions which occur between the completion of outputs and the achievement of impact.”
Of the hundreds of document available at the UNDG website, the most frequently referenced for elaboration on RBM terms are four technical briefs produced in 2007. These included technical briefs on Outcomes, on Outputs, on Indicators and on Assumptions and Risk [Update: The Word file for these briefs was removed from the UNDG website, but is still available, as of June 2018, in a cached version on Google].
The 2007 Technical Brief on Outcomes, [7 p.] the apparent foundation for many of the other UNDG documents on Results-Based Management, however, explained Outcomes at the country level this way -- The UN country teams have two separate, but linked, Outcomes at the country level:
So, for example, two Country Programme Outcomes of adoption or passage of Human Rights legislation and then adequate budgeting for its implementation might - it was hoped - lead to longer term UNDAF Outcomes of improved human rights in the country.
As the useful checklist in this Technical Brief on Outcomes noted on page 4, a Country Programme Outcome “…is NOT a discrete product or service, but a higher level statement of institutional or behavioural change.” The same checklist adds that a Country Programme Outcome should describe “a change which one or more UN agencies is capable of achieving over a five year period.”
All of this is fairly easily understandable, and as long as the assumptions underlying all of these intended results are monitored, these definitions should open the door for UN agencies to collaborate with other donors and with national governments on solid Results-Based Management, and the reporting of results. Many bilateral aid projects also have a 5-year term so it would be reasonable to see Outcomes occurring in that period.
However, this approach is not always applied, agency-by-agency, to UN results reporting at the project level. The 2009 UNDP Handbook on RBM, [221 p.] (updated in 2011) while it has many very useful components, says, of the scope of project evaluations, that the focus should be on:
The UNDP Handbook has a very good discussion of problem identification and stakeholder involvement in the development of a results framework which, it says, “can be particularly helpful at the project level” [p. 53]. But while the UNDP Handbook reiterates the importance of attention to results at the country level, this is less obvious at the UNDP project level:
The problem with this is that, if attention to results from the component parts of a development programme (i.e. the projects or activities) is missing, and if project results are not properly reported, then the foundation for country-level reporting will be, at best, hypothetical.
On the other hand, a revised draft ILO RBM Guide [34 p.] noted that:
Problems with United Nations agency reporting on results can, in part, be attributed to ambiguous definitions of Outputs.
Level of Difficulty: Moderate-complex
Primarily useful for: Anyone trying to understand the UN’s inconsistent RBM system
Coverage: 13 papers, totalling 721 p.
Most useful components: Technical Briefs on Outcomes, Outputs, Indicators and Assumptions
Limitations: Large number of potential documents laden with bureaucratic language
Who this is for
This post, and the previous review of UN agency problems in reporting on results, is intended for bilateral aid agency representatives, national government representatives, project managers, monitors or evaluators, trying to understand the inconsistent reporting of project results by UN agencies. Because the UN documents are often lengthy, and laden with bureaucratic language, it is unlikely to be of interest to people who don’t need to work with UN agency counterparts.
Background: Problems in UN RBM
This is the second of four posts assessing UN agency results chains, results definitions and problems in reporting on results, and (in the third and fourth posts) the results frameworks bilateral aid agencies use. In the previous post, I suggested that inconsistent UN agency results reporting could, in part, be attributable to a weak results culture, and sometimes weak leadership at the country level within some UN agencies. This post reviews how ambiguous results definitions also undermine UN agencies’ credibility in results reporting.
A third post in this series will review results chains for 3 bilateral aid agencies – SIDA, AusAID, and DANIDA -- define results, and the fourth and final post will review how USAID, DFID, CIDA and EuropeAid define results and results chains.
How results are defined in the UN at the country level
Language matters. I have argued elsewhere that the terminology used in Results-Based Management is dysfunctional, largely because the jargon (Outputs, Outcomes, Impact, Objectives, Purpose) can mean many different things and, that in the context of development programming, terms used for results are intended to mean something different than they mean in day to day usage. What works in almost all project contexts, however, is to focus on change as the characteristic of a result. This is a word and a concept that works in most languages, and appeals to people’s desire for common-sense terminology.
For most bilateral donors, whatever the specific terms they use, completed activities -- often referred to as Outputs -- are not sufficient for reporting purposes. While bilateral project managers are obviously required to report on completion of activities, the real emphasis in project reports is expected to be on if and how these activities are contributing to significant changes in the short to mid-term -- changes to knowledge, attitudes, policy or professional practice.
In other words, results.
Some agencies refer to these results as Outputs, Outcomes and Impacts. Others refer to them as Objectives, or Purpose, or as Immediate, Intermediate and Ultimate Outcomes. But whatever the terms, the focus is clear: “Tell us about how the project or programme is contributing to change, not just about how you spent the money”.
The problem for bilateral and national agency partners trying to hold UN partners to reasonable standards of results based management lie, I think, in the vast number of documents dealing with results in the UN context; the ambiguity of the UN definitions of results; and the confusion about how results chains for projects relate to results chains at the country programme level for different agencies.
At the UN Development Assistance Framework level, results from different agencies are essentially being aggregated, and as the January 2010 UNDAF document "Standard Operational Format and Guidance for Reporting Progress on the UNDAF" [22 p.] made clear, the UNDAF report should be “focused on reporting results at a strategic level….”. [Update: That UNDAF guide no longer appears easily accessible on the UNDAF website, but was replaced apparently, in 2011 by a longer UNDG Results-Based Management Handbook.]
Unfortunately, terms that define results for aggregations of projects at the strategic level do not necessarily work at the individual project level.
So, how did the UN, in the UNDAF and UNDG guides and technical briefs, deal with results?
UN Results Chains
Results chains describe the sequence, and the nature of links between, activities, completed activities, and near-term, mid-term and long-term results. Most practitioners agree that a direct cause and effect between activities and results is unreasonable given the wide range of intervening variables that occur in real life, but that results chains describe the general sequence of how activities can contribute to change – or results.
The several UN documents reviewed here and in the previous post variously refer to results chains (sometimes in the same document) as
Activities→Outputs→Outcomes→Impacts
Activities→Outputs→Agency Outcomes→UNDAF Outcomes→National Priority
Activities→Outputs→Country Programme Outcomes→UNDAF Outcomes→National Priority
Activities→Outputs→Agency Outcomes→Country Programme Outcomes→UNDAF Outcomes→National Priorities
“Activity Results”→Outputs→Outcome→UNDAF Outcome→National Priority
UN Agency Outputs→UNDAF Output→National Outcome→National Goal
Looking at these it is no wonder that there are differences among implementing agencies in how results are explained for projects and programmes in the UN system.
UN Outcomes
Most UN agencies use as the basis for their own definitions of results, the 2003 harmonized UNDG Results-Based Management Terminology [3 p.] which grew out of the OECD/DAC Glossary of Key Terms in Evaluation and Results-Based Management [37 p.]. “Outcomes,” the harmonized terminology states “represent changes in development conditions which occur between the completion of outputs and the achievement of impact.”
Of the hundreds of document available at the UNDG website, the most frequently referenced for elaboration on RBM terms are four technical briefs produced in 2007. These included technical briefs on Outcomes, on Outputs, on Indicators and on Assumptions and Risk [Update: The Word file for these briefs was removed from the UNDG website, but is still available, as of June 2018, in a cached version on Google].
The 2007 Technical Brief on Outcomes, [7 p.] the apparent foundation for many of the other UNDG documents on Results-Based Management, however, explained Outcomes at the country level this way -- The UN country teams have two separate, but linked, Outcomes at the country level:
- UNDAF Outcomes
- Country Programme Outcomes -- which incorporate individual UN agency Outcomes. These are not as clearly defined in these documents as are UNDAF Outcomes, but they appear to be seen as the changes to things such as policy or legislation, needed to facilitate long-term institutional or behaviour change. Bilateral donors might see these as mid-level results or Intermediate Outcomes, achievable to some degree over the period of a project.
So, for example, two Country Programme Outcomes of adoption or passage of Human Rights legislation and then adequate budgeting for its implementation might - it was hoped - lead to longer term UNDAF Outcomes of improved human rights in the country.
As the useful checklist in this Technical Brief on Outcomes noted on page 4, a Country Programme Outcome “…is NOT a discrete product or service, but a higher level statement of institutional or behavioural change.” The same checklist adds that a Country Programme Outcome should describe “a change which one or more UN agencies is capable of achieving over a five year period.”
[Editorial note, January 2012]: This document was replaced in February 2011 with the updated Technical Brief on Outcomes, Outputs, Indicators and Risks and Assumptions [21 p.], which maintains some, but not all, of the principles of the 2007 brief]
All of this is fairly easily understandable, and as long as the assumptions underlying all of these intended results are monitored, these definitions should open the door for UN agencies to collaborate with other donors and with national governments on solid Results-Based Management, and the reporting of results. Many bilateral aid projects also have a 5-year term so it would be reasonable to see Outcomes occurring in that period.
However, this approach is not always applied, agency-by-agency, to UN results reporting at the project level. The 2009 UNDP Handbook on RBM, [221 p.] (updated in 2011) while it has many very useful components, says, of the scope of project evaluations, that the focus should be on:
"...Generally speaking, inputs, activities and outputs (if and how project outputs were delivered within a sector or geographic area and if direct results occurred and can be attributed to the project)* "[p. 135] .The footnote in that quote acknowledges, however, that some large projects may have Outcomes that could be evaluated. And while later the Handbook says of project reporting that it should include “An analysis of project performance over the reporting period, including outputs produced and, where possible, information on the status of the outcome “ [p. 115] it is clear that at the project level, the priority is on reporting of Outputs.
The UNDP Handbook has a very good discussion of problem identification and stakeholder involvement in the development of a results framework which, it says, “can be particularly helpful at the project level” [p. 53]. But while the UNDP Handbook reiterates the importance of attention to results at the country level, this is less obvious at the UNDP project level:
“Since national outcomes (which require the collective efforts of two or more stakeholders) are most important, planning, monitoring and evaluation processes should focus more on the partnerships, joint programmes, joint monitoring and evaluation and collaborative efforts needed to achieve these higher level results, than on UNDP or agency outputs. This is the approach that is promoted throughout this Handbook.” [p. 14]
The problem with this is that, if attention to results from the component parts of a development programme (i.e. the projects or activities) is missing, and if project results are not properly reported, then the foundation for country-level reporting will be, at best, hypothetical.
On the other hand, a revised draft ILO RBM Guide [34 p.] noted that:
“Some mistakenly think that outputs are ends in themselves, rather than the various means to ends. RBM reminds us to shift our focus away from inputs, activities and outputs—all of which are important in the execution and implementation of work—and place it instead on clearly defined outcomes....” [p. 17][June 2018 update: This draft is no longer available, but a new ILO RBM and M&E Manual was produced in 2016.]
