Evaluability Assessments and Results-Based Management: 8 Guides

by Greg Armstrong

Last Update:  July 5, 2022

Evaluability Assessments answer the question:  Is there enough available information to justify the time and cost of a full-scale evaluation? If they are done early enough during implementation, they can identify basic problems in design which can guide remedial action.

One old study set the stage for 8 recent detailed Guides which synthesize earlier work and provide useful advice on whether, when and how to undertake an evaluability assessment – or to use other approaches to assessing project or programme design integrity.

Evaluability Assessment Guides

Level of difficulty: Moderate
Length: 3-72 pages
Primarily useful for: Evaluation managers, RBM specialists doing evaluation assessments
Most useful: DFID Guide to Planning Evaluability Assessments
Most interesting:  Program Management and the Federal Evaluator (1974)
Limitations:  These guides tell us what needs to be done, but they require people with the process skills to do it all.

The History of Evaluability Assessments

Evaluability assessments, pre-assessments or exploratory evaluations, as they were known in some cases 40 years ago, have been used for many years in public health research where the term “evaluable” refers to ‘Patients whose response to a treatment can be measured because enough information has been collected”,  education,  and justice  (PDF) and the term has academic antecedents in testing mathematical propositions, far back into the 19th Century. 

The Foundation Evaluability Assessment Document

The foundation evaluability concepts for social programmes were initially presented, from what I can see in a 1974 Urban Institute study, one of many emerging during implementation studies of the Johnson administration's Great Society Program,  published as "Program Management and the Federal Evaluator" in the Public Administration Review, included in a 1977 volume of Readings in Evaluation Research, available through Google Books. 

The Foundation Evaluability Assessment article

It advocated looking at issues which are at the heart of current guides to evaluability assessment - and for that matter, at the heart of solid results-based design:

• Whether there is a clearly defined problem addressed in design
• If the intervention is clearly defined
• If the short and longer term results are defined clearly enough to be measurable
• if “the logic of assumptions inking expenditure of resources, the implementation of a program intervention , the immediate outcome to be caused by that intervention, and the resulting impact” are “specified or understood clearly enough to permit testing them”.
• Whether managers are capable of and motivated to using performance data for concrete management decisions.

While there are many other guides to evaluability assessment which are much more detailed of more practical utility today, the most important core concepts are here in this 1974 study, and this short, 43-year old article is still worth reading for its insights on clarity on language, assumptions and results.  

Leonard Rudman’s later 1980 book Planning Useful Evaluations - Evaluability Assessment  set out a detailed approach to dealing with all of these issues.

Changes in Evaluability Assessment Utilization

Evaluability Assessments started out as ways of improving the design of evaluations, to increase the chances they will be useful to the people who are funding and implementing activities, and in recent years, with a substantial increase in the number of evaluations of international development projects, many agencies, including UNICEF, the World Bank, UNDP and numerous bilateral donors have incorporated evaluability assessments as part of the project cycle, in theory during the design process, and after design, during implementation and before a decision is made to pay for a full-scale evaluation.  

Many Evaluability Assessment Guides have been produced in recent years, but here are just a few worth noting.

A Useful Introductory RBM Guide for Long-term Planning

Greg Armstrong

Updated August 2019

Monitoring and Evaluation: An Approach to Strengthen Planning in Cambodia provides a useful introduction to the application of RBM in planning.

Level of difficulty: Moderate
Length: 55 pages
Primarily useful for: Those new to the complexities of the use of RBM in long term planning
Most useful:  Detailed guidance on results, indicators and targets at different levels of government operation, p. 26-45
Limitations:  Some of the links have expired and the document can be difficult to find on the Cambodian government website.

Strategic Planning

We rarely see national strategic plans, 5-or 10-year plans in well-established, institutionalized democracies with a diverse population, because this diversity is reflected, in a functional, institutionalized democracy, in its electoral outcomes.  As we have seen recently, unless long-terms plans are built on a consensus on values and methods, sustaining policy change will be difficult in the face of changing electoral results.

But organizations in all countries, where the purpose is more focused and diversity within the organization is more limited, often do undertake such plans, and we can see those in a host of strategic plans for health, education, food security, gender, environmental sustainability, transportation and other areas, in both international aid agencies and within organizations in individual countries, where at the national level planning itself may be the chaotic byproduct of democracy.

Where long-term national plans are undertaken,  it is often in countries with a legacy of central state planning, and in many cases, I suspect, these plans are developed as a tool to explain to external funding agencies how money can be usefully applied to problem solving, and how a country’s strategic plan is compatible with an aid agency’s long term priorities.

Such planning is obviously very complex, and finding the tools to help bring some order to what could be a chaotic and intimidating process, is important.  It is here that results-based management can be particularly useful. It will not shorten a planning process, but it will make it more rational, provide a productive path to follow, and, if applied intelligently, will increase the chances for the achievement of results.

The Cambodian guide to integrating M&E in long-term planning

In 2012, the Royal Government of Cambodia established a National Working Group on Monitoring and Evaluation to consider how M&E could be usefully applied and integrated with the country’s 2014-2018 planning cycle.  Sarthi Acharya, working with this group as a UNDP advisor, produced this summary guide to how the major M&E concepts – all of them important elements in results-based management – could be used to facilitate and assess the planning process.

What makes this a useful tool is not its originality.  As the document notes itself notes:

“This is not a research paper. It is a contextualised primer and its main audience are policy makers and programme evaluators in the Royal Government of Cambodia. The reader is expected to look for the meaning and application of the concepts and approaches presented here rather than search for originality in the research.”  [p. 5]

While some – but not all – of the illustrations and tables have been derived from other existing work and sites, such as Tools4dev,  The Monitoring and Evaluation News   or from academic sources or aid agency RBM guides, the references to all of these are provided in this guide for those who wish to go deeper.  Some of these have links to original documents, and not all of the links still work, because links often expire as documents move. It is possible, however, to find many of the original documents cited here, online and I have provided links to many of these, and other RBM guides and M&E handbooks at the end of this post.

Many of the most interesting parts of this guide are, in any case, original illustrations of how concepts could be applied specifically to the target audience for this document.

Theory of change applied to Cambodian issues
Click to enlarge

Those who want a comprehensive  discussion on establishing an RBM system can read Kuzek and Risk’s 247 page  10 Steps to a Results Based Monitoring and Evaluation System, published in 2004 by the World Bank.  And for those who want a hands-on step by step detailed guide to project planning, Peter Bracegirdle’s 2001 RBM guide  A Results Approach to Developing the Implementation Plan  is still useful.

This Cambodian planning guide, was never intended for wide distribution, and although it can be downloaded (for now) from the link at the top of this post, it is buried deep in the Cambodian Ministry of Planning website.  But it could be a useful tool for many policy makers and managers in other countries, people who do not necessarily have the time, or the inclination to go to original sources, or to spend time on the detailed guides cited above. It will be useful to professionals and managers who just want an overview of what the major elements of results-based management are, in the development of long-term plans. The examples used to illustrate the processes are, certainly, given its purpose, specific to Cambodia, but present issues and potential programmes in fields such as school education, water and sanitation and rural poverty alleviation which arise in many countries.

And when it comes to how to organize the presentation of results indicators linking the broader aspects of RBM - national goals, policies and actions to a specific programme structure, the guidance here is detailed enough to be of practical value.

Templates for aligning National and Ministry Results
[click to enlarge]

The guide provides a number of templates for the organization and alignment of results and indicator data at each of these levels, accompanied by guidance on how to complete the templates


The point here is that the target audience, while primarily Cambodian, is also clearly people who want to know how to fit different approaches to RBM together, people who want something practical, which they can apply, something which will introduce them to, or remind them of how different elements of results-based management can be used.

The document has what appear to me to be three basic sections, the first two of which are likely to be of the most utility to planners in other countries:

• An initial 25 page overview of basic issues in RBM and planning, with some examples from the Cambodian situation on pages 1-26
• An interesting set of guides and tables to facilitate systematic indicator data collection, analysis and reporting pages 26-45
• A concluding 8-page section discussing the specific indicators and results in the 2014-2018 M&E framework for the Cambodian National Social and Economic Development Plan. 

An Overview of Basic RBM Tools

Comparing a results framework and a logical framework
[Click to enlarge]

The first 25 pages of this guide provide introductory summaries on a number of topics:

• How log frames, results frameworks (and what some agencies call logic models or conceptual models) relate to each other
• The Theory of Change – and its relationship to log frames and planning
• Very brief case studies of how a theory of change could be used to assess problems in education, sanitation and poverty alleviation in using examples from Cambodia and Laos
• The links between programme structures and results frameworks
• A summary of how a national plan can incorporate work at the Ministry-level, the programme,sub-programme  and activity level
• A short summary of differences between monitoring and evaluation.

Detailed guidance on organizing results, indicators and targets

More detailed is the document’s  guidance, in pages  26-45 on the approach used in Cambodia to organize and present data for intended results, indicators, and targets and indicators at 4 levels of government operation within the national plan.

The casual reader might think these will be of only limited interest, because of their specific links to Cambodia, but, this approach – using tables to organize the reporting, or variations on it could be adapted for use elsewhere.

Included here are tables and guidance on:

• National goals, planned actions, macro indicators and targets, with notes on issues in data collection
• Ministry level goals, indicators, targets, achievement, budgets  and actual expenditures
• Programme-level objectives - and how they link to Ministry and national goals,  indicators, targets, achievements, budgets and actual expenditures
• Sub-programme objectives – and how they link to the programme , number of activities, indicators, targets, achievements, budgets and actual expenditures
• Activity-level objectives, how they link to the sub-programme, with indicators, targets, achievements, budget and actual expenditures, for each activity, and spaces for comments on issues related to gender, environment, income distribution and technical details of the activity.

Formats for planning and monitoring results, indicators and targets
Click to enlarge

Anyone who has to contend with reports which begin with discussions of the activities, burying actual results information far down in often inaccessible tables and meandering text will appreciate this approach to putting results up front, and leaving the details on activities for later descriptions, (for those who are interested).  It is not always necessary to use formal tables such as those presented in this guide, and the groups I work with often end up using a similar approach in text.

Process and Outcome Evaluation template

[click to enlarge]

But people’s attention to detail on indicator data collection, organization and presentation often deteriorates over time with the competing pressures of day to day implementation, and for an enterprise as large and complicated as the Cambodian national plan, the tables are probably necessary to get any consistency of data at all across multiple ministries, programmes and activities.

In the context of this ideal vs real world implementation of RBM, it is interesting, but not completely surprising to me,  to note that the actual National Strategic Development Plan produced subsequent to this guide in 2014 does not have this level detail.  There are a large number of national and sector indicators in the plan, but none I could find at the Ministry or subprogramme level.  As the chapter on Monitoring and Evaluation notes:

"There are several capacity gaps related to M&E in all line ministries and agencies.

• Line ministries and agencies do not have adequate capacity to formulate SMART(Specific, Measurable, Achieveble, Realistic and Time-bound) indicators for their sectors.
• Line ministries and agencies do not have adequate capacity to collect and analyse data for measuring their indicators.
• The RGC does not have an National M&E System to monitor and evaluate the progress of implementation of NSDP and the implementation of all projects carried outby line ministries and agencies in the Three-Year Rolling Public Investment Plan." [p. 218]

M&E Orientation Guidelines for the NSDP produced a year and a half after the plan itself was published make it clear that these problems had to be addressed.

The existing Results Framework for NSDP implementation requires additional work to be able to assess performance. Even though it includes indicators, baselines,intermediate and final targets, it does not include indicators of efficiency (value for money), cause and effect relations to establish the contribution path, or an indication of the evaluation agenda. This will require an articulation of various instruments at planning, programming and budgeting levels at institutional level and across institutions; as well as an enhanced ability to collect relevant, timely,and accurate administrative data to support the analysis for reporting.[p. 6]

These M&E Guidelines suggest that work was planned to develop the capacity of Ministries to collect and report on the kind of data suggested here.  It will be interesting to see if this actually occurs, or if this level of application of RBM to planning, will have to await subsequent plans.

The bottom line:  Although it remains to be seen if the RBM tools presented in this guide will be applied in action in Cambodia, this document provides a useful introduction to basic RBM tools for planning, and some practical guidelines on data presentation, which could be applied not just in national planning but in programme and project planning in many countries.

Related Resources on Results-Based Management and M&E

• Better Evaluation 
• DFID's Review of the use of ‘Theory of Change’ in international development 
• The Monitoring and Evaluation News  [see review]
• The WWF's Basic Guidance for Cross-Cutting Tools: Conceptual Models, Stakeholder Analysis, Situation Analysis and more
• Actknowledge publications
• RBM Websites
• The University of Wisconsin's online course on developing logic models.

_____________________________________________________________

GREG ARMSTRONG

Greg Armstrong is a Results-Based Management specialist who focuses on the use of clear language in RBM training, and in the creation of usable planning, monitoring and reporting frameworks.  For links to more Results-Based Management Handbooks and Guides, go to the RBM Training website.