Using Logic Models in Results-Based Management

--Greg Armstrong --

[Links updated 2018]

This website houses a large number of articles, some of them quite complex, on how to construct logic models, using proprietary software.

The Outcomes Theory Knowledge Base
Level of Difficulty:  Complex

Primarily useful for:  RBM specialists, academics
Length: 50-60 web pages
Most useful sections: Articles on evaluation

Designing Logic Models - Review by Greg Armstrong

The “Outcomes Theory Knowledge Base” is the title given to a compilation of more than 50 articles on what the author refers to as Outcomes Theory --  what many of the rest of us refer to as RBM, or management for development results. Most of the articles focus on how to use visual Logic Models for project management.


Who this is for


Readers will need to sift through 50+ articles, all written by Paul Duignan, to find what they need. But, although there is a lot of repetition in many of the articles, some of them could be useful to three groups:

• Those interested in learning how a visual approach to results, through the development of logic models or outcome models can clarify results discussions.
• Those who want an overview of some broad issues in evaluation.
• Those people interested in an academic analysis of how results are viewed in a broad conceptual format.

For most field project managers, and host-country counterparts, the utility of many of the articles will be limited by the relatively dense language used to explain some common-sense ideas (for example, see the article “Problems faced when monitoring and evaluating programs which are themselves assessment systems”. Some simpler summaries on logic model development are also, however, available at a related commercial website, EasyOutcomes.org. 

The Utility of a Visual Logic Model

While there is considerable overlap in the ideas discussed in the more than four dozen articles originally included in what Google refers to as a “Knol” or a “unit of knowledge”, the reader who takes the time to work through these will find some useful material.


By my estimate at least 30 of the articles focus on the advantages to project managers, evaluators and monitors of using a visual approach to managing for results - Outcome Models, Logic Models or other visual representations about the relationship between activities and results. The core of these articles (although each puts these in a slightly different context) is based in some common-sense ideas that many RBM trainers, planners or evaluators may recognise from their own experience. Among these is that in planning, monitoring and evaluating for results, we should:

1. Focus on results, not activities - and label results as “Outcomes”.
2. Use a visual logic model to clarify results. This makes it easier to see the relationships between activities and different levels of results than is possible using a Logical Framework.
3. Distinguish, in the logic model, between a) results and indicators for which an agency is directly responsible in the near term, and b) higher level results, for which attribution to the intervention (for success or failure) is not clear.
4. Hold managers responsible for two primary tasks: a) Achieving results for which there are clear indicators and a reasonably clear and accepted causal relationship between activities and results; and b) Managing for development results at a higher level, in part by collecting and reporting on indicator data on results for which there is less certainty of attribution.
5. Frame contracting, monitoring and evaluation within the context of the results, activities and indicators identified in the visual logic model.

Most of these articles also suggest that the proprietary software (DoView)  sold through a related website, can help us do all of these things more efficiently and creatively than we can by relying just on tables in word processing software. Taken together these four dozen articles also appear to form a help file for those using that software.

Evaluation issue summaries

At least ten of these articles focus specifically on evaluation. While the author obviously thinks that the visual logic model would assist in focusing evaluation questions, the articles go beyond this, and some provide what could be, for those looking for quick summaries, useful overviews of major evaluation issues.


Among those articles that could be useful to readers, whether they use the author’s software or not, are:

• “Impact/outcome evaluation designs and techniques” which discusses in broad terms seven different research approaches to impact and outcome evaluations;
• “Impact Evaluation - When it should and should not be used;
• “Distinguishing evaluation from other processes (e.g. monitoring, performance management, assessment, quality assurance)”. 

Greg Armstrong’s analysis

Key Resources on Evaluation and RBM?

While many of the articles on the logic model and on evaluation are useful, the article on “Key Outcomes, Results Management and evaluation resources” provides fewer useful links than the average reader might expect, from someone of the author’s experience.


The descriptive summary says it contains ”A summary list of key outcome theory related resources for working with outcomes, results management, evaluation, performance management, outcomes-focused contracting and evidence-based practice”.


“ Ahha!” I thought, “just what people who want to learn about RBM should have - ideas from the UN, DfID, CIDA, SIDA, Universities, government agencies, think-tanks, trainers and NGOs.” This could have been very useful to professionals seeking user-friendly tools on evaluation and RBM.


A quick review, however, shows it contains, at least at this writing in 2010, just 14 links - all of them to one of 8 of the author’s own websites, including his blog and twitter feed, and all with links to the sale of the logic modelling software. The author obviously has a history of work in evaluation, and presumably knows of other useful sites.  


Links to other relevant sites, such as, for example the Monitoring and Evaluation News, or the Centers for Disease Control's Evaluation Working Group resources would have been helpful to people looking for useful tools. 


The list of references to Outcome Theory  similarly lists 33 articles, all written by this author. Many of them are probably useful, but a broader net might have brought in ideas about the work other people have done on similar or related topics.

The Value of Logic Models

While there is some overlap in the content of these different articles, the basic point being made here is valid - using a Logic Model diagram as the focus for discussion can, indeed, as I have found recently in workshops in Vietnam, Cambodia, Indonesia and Thailand, clarify differences of perception over results, assumptions about cause and effect, and can energize discussions on project design and evaluation.


In one of the articles on this website, dealing with the value added of evaluation, to governance and policy making, using a visual logic model, the author writes:

“Outcomes models need to be able to be used in all parts of the decision-making process. In order for them to be able to be used in this way, their visualizations needs to be portable across different media so that they can be used whenever and wherever they need to be used. For example, they should be able to be developed and used in real-time during meetings with high-level stakeholders, printed out in a report, and reproduced on an intranet or the internet. Meeting this criteria requires using appropriate software and laying out an outcomes model in a way that ensures that it is portable”.

Software Limitations for Logic Model Development

I facilitate workshops on developing results frameworks, logic models and indicator assessment several times a year, in almost all cases in countries where English is at best a second language, where internet access is often unstable, and in some cases where electrical power is unreliable.


I have not used the DoView software which many of these articles are linked to, in such workshops but I can see from its description that it could be helpful particularly during facilitation of logic model development workshops. Given that this software was developed particularly with results chains in mind, it could possibly have an advantage over other visual mapping software of a similar nature, such as Xmind, Vue or SmartDraw among many others. Like those other programmes, too, however, the software promoted here has limitations which would diminish its utility for facilitators working in the situations I work in.


At the end of a Logic Model development workshop, one important deliverable is a draft Logic Model and possibly an indicator assessment framework, which the users can take back to their many different offices, in different countries, different provinces or cities, a document they can distribute widely to their own colleagues and their own networks, for further critique and possible alteration.


The price for the DoView software is not high - roughly $35 per copy, cheaper than others that can run to several hundred dollars - but not as cheap obviously, as Vue, Xmind or others that are free. Even the free programmes have a problem with accessibility and portability however. Having used any of these programmes to engage people in a dynamic discussion of results, what do you do next, when they want to continue the discussion with their own partners? Do you ask them all to download and install the programmes?


It is not clear to me that the Logic Model diagrams from any of the visual mapping programmes I have seen can actually be edited with standard, commonly - available word-processing software such as Microsoft Word, OpenOffice Writer, or Google Docs. While Logic Models produced with DoView, Xmind, Vue, SmartDraw and many other similar programmes can be viewed using those programmes, or alternatively in PDF or on the web, and can be pasted into word processing programmes, in most cases they cannot be edited by people who do not have the original software in which the diagrams were originally produced -- making downstream participation very difficult.


For those programmes which are web-based, some editing can be done on the internet, but accessibility does not rest in “the cloud” for people where internet access is not always reliable.  The bottom line is that the utility of all of these mapping and diagramming programmes is limited where it is impractical to install specialised programmes on dozens of different computers.


If portability really is the criteria for assessing all of these programmes, then the priority should be not just the ability to view the results in a PDF file or on the internet, but the ability of partners to critique and edit the models.

No Painless Performance Indicators

Another issue, is that none of these programmes, with these limitations, will be easy to link to the other half of the results discussion - in many ways the most time consuming portion of results-based planning -- the assessment of the utility of indicators.


As anyone who has worked through the indicator development process knows, it can take days for project partners, working in groups, to sort through potential indicators, testing them for validity, for the existence of baseline data, for the availability and accessibility of reporting data, for the existence of appropriate research skills, and the time required for data collection and analysis.


While several of the articles in the Outcomes Theory Knowledgebase refer to the tongue-twisting “Non-output attributable intermediate outcome paradox”  and make a reasonable point about attribution, none of them makes the job of assessing indicators any easier, any faster or any more accessible for partners.


The Outcomes Theory Knowledgebase web site has many “how to” videos, hosted on YouTube, aimed primarily at helping people use the proprietary software. One of these is titled “Painless Performance Indicators: Using a Visual Approach”. This got my hopes up!


But, foiled again: What the video demonstrates is that if you have already done all of the hard work on indicators, having completed this assessment, you can insert a reference to the existence of the indicator, in the Logic Model, using the software. I am sure this is useful (although it can also be done with word processing programmes and the use of hyperlinks) but the point is that inserting indicators in a visual model is not the painful part of indicator development.


For the time being, until something new develops, I will be sticking with the basic word processing programmes which allow a facilitator to work with participants to develop a logic model (albeit without some of the ease of the mapping software) and then link and integrate it with an indicator assessment worksheet, as indicators are being proposed, tested, rejected, modified and accepted. But, I continue to live in hope, and may revisit the issue of software again later.


The bottom line: "The Outcomes Theory Database" includes articles with some useful arguments in favour of using a visual logic model approach, and some quick summaries of evaluation issues, but there is no magic bullet here.

Other resources on Logic Models:

• The University of Wisconsin's detailed and engaging online self-study module Enhancing Program Performance with Logic Models

• The W.K. Kellogg Foundation's Logic Model Development Guide.

• Using PowerPoint to develop Logic Models, from the Centre for Community-Based Research.

_____________________________________________________________

GREG ARMSTRONG

Greg Armstrong is a Results-Based Management specialist who focuses on the use of clear language in RBM training, and in the creation of usable planning, monitoring and reporting frameworks.  For links to more Results-Based Management Handbooks and Guides, do to the RBM Training website. 

Applying RBM to Policy

--Greg Armstrong -- 

[Edited to update links and content August 2016]

Can policy making and policy advice be assessed by the same performance assessment standards and RBM methods that are applied to programmes? Mark Schacter’s views have evolved over the past five years.


Level of Difficulty:  Moderate to complex
Primarily useful for:  Policy makers, performance-management divisions
Length: 2 papers, 42 pages
Most useful section: What constitutes “good policy advice” p. 7-8 in “The Worth of a Garden”
Limitations:  Focused primarily on senior officials, unlikely to be useful to field workers, unless they are in a policy development project.

Who this is for


Senior policy makers and performance management officials in the public service will recognise the issues Mark Schacter raises in all of his writing on RBM. Targeted primarily on Canadian public officials, the issues he raises are relevant to public servants everywhere. The policy discussions, however, are less likely to meet the needs of development field workers, or project managers, unless they are working specifically on policy development projects.


A decade of RBM analysis


Mark Schacter has been working on results-based management, training people on performance measurement and writing about it, since at least 1999. While this is certainly not the only thing he does his writing on RBM has been prolific, and influential.   The CIDA/Global Affairs Canada revised RBM terminology, formally adopted in 2008, for example, bears a striking resemblance to that used by Mark Schacter in his 2002 paper “Not a toolkit: Practitioner’s Guide to Measuring the Performance of Public Programs”


First at the Institute on Governance, and more recently as a freelance consultant, he has published [at least 40] articles since 1998, focused on performance measurement and RBM. Some of these can be found on the Institute on Governance web site, but more are available directly on one of his own web sites: Mark Schacter Consulting.

[Editorial note March 2016: This review does not cover a number of new articles which can be found at the Mark Schacter publications page.]

Should Policy be judged by the same RBM standards as Projects or Programmes?

While the paper “Not a Tool Kit” provides a useful summary of the steps and issues in RBM for the public service, particularly the discussion of tradeoffs on indicators, the focus of this review is on how he treats policy making in the RBM context.


Two of his articles demonstrate how Mark Schacter's views shifted between 2002 and 2006, although I think, only marginally, on the important issue of whether standard performance measurement processes can -- or should -- be applied to policy development and public servants’ role in providing advice.


Is Policy Unique?


A strong advocate of intelligent application of RBM to the management of public programmes, in his 2002 article, What Will Be, Will Be The Challenge of Applying Results-based Thinking to Policy, he reviewed the standard arguments against applying results-based management to policy -- that policies are intangible things, subject to a variety of influences, such as politicians’ short-term political needs, that there is often a huge lag in time between policy development and any chance of seeing concrete results. The conclusions, for those who take this view -- and I have heard this recently -- is that policy is therefore “unique”, and that tracing the effect of advice on the success of policy is therefore in some way, “unfair”.


Schacter took the view, in this 2002 article, that intangibility and complexity are not unique to policy but occur often in programme implementation. He appeared to take the view that while these things present challenges to people attempting to assess performance, they also provide an opportunity to use performance measurement, and the critical examination of a logic model, to clarify assumptions and test the understanding of the intended results, implied or clearly stated by policy.

Evaluation and Performance Measurement

The link between the views Mark Schacter held in 2002, and those he expressed in 2006, is the role of evaluation in assessing the effectiveness of policy. Performance measurement, he wrote in 2002, looks at where we are today, and tries to assess how likely we are to achieve long-term results, by looking for evidence that we are making progress against shorter-term results. 

Evaluation, on the other hand, assesses not just whether results have been achieved, but whether they were the most appropriate results, why results were, or were not achieved, and whether alternative means of achieving them would have been more appropriate. [p. 17]

[2016 Edit:  In 2011 Schacter added a new paper - Tell Me What I Need to Know: A practical guide to program evaluation for public servants, which makes some useful distinctions between the policy requirements of monitoring and evaluation.]

The case for performance measurement of policy

Performance measurement, he wrote in 2002, has its limitations, particularly given the usual lag between policy development and achievement of long-term results. But, he continued:

“Sometimes a less-than-perfect instrument is, under the circumstances, the best one for the job at hand. Performance measurement is indeed a “second-best” instrument – but a very useful instrument nonetheless….Citizens have no less a right to be informed about the performance of policies than of programs. In order to explain and justify the allocation of resources to …any policy (or program) you need to have a way of connecting what you are doing now with where you want to be in the long term. This connection needs to be clear and must make sense not only in the minds of the people responsible for the policy, but also in the minds of external stakeholders (citizens, civic groups, private sector operators, politicians, etc.).

Performance measurement helps you make that connection. It helps you tell a believable and compelling story about why a policy was conceived in the first place, and whether or not it appears to be on the right track.” [p. 24-25]

The case for evaluation of policy

By 2006, in a paper for Canada’s Treasury Board, “The Worth of a Garden: Performance Measurement and Policy Advice in the Public Service” Mark Schacter had apparently come to the conclusion that measuring policy performance in the short term, might in fact be too challenging, and that an emphasis could probably be more productively put on longer-term evaluation.

He outlined two options for using performance measurement on a regular basis to assess progress towards policy results. The first is to assess what he called the “process and outputs standards for policy advice”, the second is essentially what he advocated in his 2002 article - to assess progress toward achievement of immediate and intermediate outcomes - whether policy advice was accepted and implemented. The conclusion he came to in 2006, however, differed from his earlier view:

“Low-specificity organizations and tasks pose especially difficult problems for performance measurement – problems so significant that it may be impractical (if not impossible) to apply standard performance measurement in a way that yields useful results. This does not mean that one should not attempt to assess the quality of a policy shop’s performance. But it does suggest that evaluation may be worth considering as a better tool than performance measurement for this particular task. Evaluation, though closely related to performance measurement, differs from it in ways that may provide a better fit with the subtleties and ambiguities of the policy-advice process.” [p. 11]

Greg Armstrong’s comments:

Are performance assessment and evaluation mutually exclusive?

At no point did Mark Schacter advocate abandoning the assessment of policy units’ performance. He has always maintained that at some point policy functions have to be assessed.


What is unclear to me, however, is why the assessment options -- regular performance assessment and eventual evaluation -- appeared to be regarded as mutually exclusive. [2016 edit: His 2011 paper - referenced above, sees them as complementary elements on an "evaluation continuum".]

It seems to me that combining a) an assessment of the quality of policy advice, and the processes which lead to it, with b) an assessment of interim results, and c) a longer-term evaluation, is a reasonable (if obviously not perfect) way of helping policy advisors, policy makers, legislators, and those who fund them, to understand the progress they are making toward long term results.

By 2008, Schacter was writing about other performance assessment issues, and in How Good is Your Government: Assessing the Quality of Public Management [2008] policy was mentioned only once, in passing. One type of information he proposed in that article, for assessing the efficient management of resources, however, was “Results-based performance information is used routinely as a basis for continuous improvement of program/policy performance.” [ p. 5] 


This suggests that he had not given up completely on the contribution to the policy function of regular performance assessment.

How RBM applies to Policy Projects in International Aid

It is important to note that none of Mark Schacter’s writing, at least between 2002-2006, was focused on whether performance assessment could be applied to improved capacity to provide policy advice. 


If, as I contend, there is a role for performance assessment in assessing progress on policy in general, there is surely a much clearer role for it, and for RBM in general, in planning, implementing and assessing results for international aid projects which focus on the development of capacity for policy research, policy formulation, and legislative capacity.


Mark Schacter maintained in 2006 that the provision of policy advice is essentially an output - a completed activity. But while there could be a case made that this is true for some policy functions, in the context for which he was writing -- and even that is not completely clear to me -- it is not true for policy capacity development. Improved quality of the policy making process, and improved quality of the advice provided, are both clearly interim results in capacity development terms, and therefore worth assessing on a regular basis.


In the 2006 paper, Schacter outlined the commonly regarded criteria for assessing the quality of the policy advice process, adopted in part from studies in Australia and New Zealand:

• The timeliness of the advice for decision-makers
• Relevance of the analysis to the current realities faced by decision-makers
• Stakeholder consultation underlying the proposed policy
• Clarity of purpose (essentially - does the policy itself rest on a solid logic model)
• Quality of evidence, and the link between evidence, policy and purpose
• Balanced range of alternative and viewpoints reflected in the analysis
• Presentation of a range of viable options
• Clarity in presentation
• Pragmatic assessment of the potential problems of implementing the policy.

All of these, with some work, could form the basis for useful performance indicators for policy capacity projects or programmes and have, in many cases, been used for this purpose. Certainly as Mark Schacter observed, indicators relevant to these issues would provide qualitative data -- subjective in nature, and time consuming to collect.


But, in my experience, qualitative data are not necessarily any more time consuming to collect than quantitative data, and certainly not less valid if the intention is to assess the quality of the policy formulation process.

The bottom line:

Policy development is, indeed, sometimes an uncertain process, but there are ways of improving it, of building capacity and of assessing this capacity. Mark Schacter’s articles on the role of performance assessment in policy clearly outline the challenges, but also deliver some reasonable suggestions on how to deal with them.

Further reading:

• For a useful discussion of the indicator tradeoffs between accuracy and practicality, see Not a “Tool Kit”: Practitioner’s Guide to Measuring the Performance of Public Programs, [2002] pages 25-27 and particularly the figures 8-11, p.41-45.
• For a summary of the arguments against, and for using RBM at all in the public service, see Trashing Results-Based Management or Throwing Out the Baby with the Bath Water [2006].
• For an attempt to create a performance assessment framework to be applied to whole-of government performance, see How Good is Your Government? Assessing the Quality of Public Management [2008].

[2016 - Other very useful more recent papers can be found on Mark Schacter's website, including several on evaluation, monitoring, the use of performance dashboards and risk assessment.]

_____________________________________________________________

GREG ARMSTRONG

Greg Armstrong is a Results-Based Management specialist who focuses on the use of clear language in RBM training, and in the creation of usable planning, monitoring and reporting frameworks.  For links to more Results-Based Management Handbooks and Guides, go to the RBM Training website. 

Indicators: a simple analysis

-- Greg Armstrong -- 

The Ants and the Cockroach: A challenge to the use of indicators,  and A Pot of Chicken Soup, A response

 by Chris Whitehouse and Thomas Winderl; and Rick Davies’ commentary


Aesop’s Fables explains indicators. A  deceptively simple and engaging introduction to some complex arguments on indicator development and use. Simple metaphors outline the case both against, and for the use of indicators in results-based management.


Level of Difficulty:  Simple to moderate
Primarily useful for:  Anyone who wants to review the basic arguments on indicators
Length: 12 pages, combined
Most useful sections: Engaging metaphors for RBM
Limitations:  Too short to guide practice


Who this is for


Anyone who wants an introduction to what the past ten years of debate on indicators and RBM are about, or anyone who is fatigued with more sophisticated analyses and  just wants a laugh.  Trainers might find this useful too when the usual RBM exercises fade.

Simplifying the indicator debate

Development professionals know how complex the process of developing good indicators can be.  Doing it effectively - or at least in a manner that will give you a fighting chance of producing information relevant to how a project is performing - requires the major participants in project planning and implementation to work together.  Exploring the validity of an indicator, its political implications, and the practicality of data collection, can take a long time and can be, as evaluators say, challenging.


It is not that the process of defining usable indicators is necessarily intellectually difficult, but it does require sustained attention.  There are so many questions that need to be answered as you weed out the useless indicators, that the process invariably takes time, and patience.


While answering them may be time consuming, the basic questions that need to be asked in assessing indicators, are simple, something that often gets lost in more complex analyses.


In 2004, Chris Whitehouse, who was a United Nations Volunteer Programme Officer in Bhutan, took on his colleague, Thomas Winderl, who was the UNDP’s Head of the Poverty Unit in Bhutan, in what looks like a good-natured debate on the utility of indicators. The result is this short combination of two points of view, apparently simplified to the very basics. I say “apparently” because a careful reading of the articles reveals the complexities the authors were obviously aware of, when they produced these papers. This  particular article appeared on  the Monitoring and Evaluation News website, in 2004, and the exchange was later included in an anthology of articles on monitoring and evaluation, published in 2006, titled: Why did the Chicken Cross the Road  


Other critiques of the LFA and RBM

The core of the arguments against indicators and the LFA were not particularly new at the time.  The article appeared several years after Rick Davies and Jess Dart had begun working on an alternative to the use of the LFA, at about the time they were writing The Most Significant Change Guide. The Outcome Mapping Approach had also been developed largely by IDRC, roughly 3 years earlier, and was also starting to gain some momentum. This was also at a time when interest was just beginning to appear in what is now referred to as “impact evaluation” or more confusingly as “counterfactual” analysis.


The debate on the use or misuse of indicators, the LFA and RBM,  has continued since 2004, but never in such a user-friendly format as was the exchange between Chris Whitehouse and Thomas Winderl.

Challenging the validity of indicators

Chris Whitehouse introduces what he sees as some basic issues in indicator development --

• whether the indicators are measurable,
• who will collect the information,
• cost and baseline data.

But the potentially dry discussion is made more engaging through the metaphor he uses - the stakeholders are millions of ants, the project involves first, the movement of a dead cockroach to their nest, and second, after learning lessons from that experience, moving a dead beetle to the nest. The project manager is the Queen ant, persuaded, perhaps against her better judgment, to measure the effectiveness of individual and group progress on getting the cockroach to its destination.


Assigning 10% of the work force to monitoring, a host of different indicators are tested, data collection becomes chaotic, the workers are distracted from their task, focus obsessively on indicator data collection, and decisions that will achieve the indicator, but ignore the underlying long-term result - more food.  At the end of the  whole sorry process, nobody knows anything useful about the result, but they have learned a lesson about the perils of using indicators.


Interspersed with the saga of the cockroach is the less engaging, but more realistic discussion of the same issues as they might appear in a UNDP project focused on providing computer training for civil servants.  Together the two stories make the points that are now familiar to anyone who has been following attempts to make the logical framework approach more usable, or to develop alternatives to the approach:

• Technically valid indicators are often impossible to measure
• Measurable indicators are often trivial or misleading
• Good indicators can take so much time and money to develop, that they interfere with more important work
• Focusing on indicators can bias programming decisions to the exclusion of more useful activities that might not produce measurable indicator data.
• The use of indicators implies cause and effect, a scientific validation for activities - and without control groups, this is not valid.

Chris Whitehouse’s conclusion is that the logical framework approach is primarily useful as a tool to test the logic and assumptions implicit in project design, but the use of indicators misrepresents and ultimately undermines that simple utility.

Defending the use of reasonable indicators

Thomas Winderl replied with the analogies of cooking soup, and trying to figure out what the weather is.  What indicators do we have that will tell us when the soup is ready to eat? What indicators tell us whether we should wear an overcoat or take an umbrella on a walk?


The soup is relatively easy - watch it, does it boil?  Is it too hot to eat?  Too cold?  No need for complex equipment, or outside expert monitors here.   Similarly, if we look out the window we will have some fairly useful indicators about the weather?  Is it snowing, raining?  What are people wearing?  Shorts?  Gloves?


The points he makes are, again familiar, but well summarised here:

• Money spent on indicator development and monitoring  should be proportional to the overall project -- but in fact most project vastly underallocate resources to monitoring and evaluation.
• Monitoring bad or poorly developed indicators is a waste of time but putting  reasonable time and money on indicator development and monitoring is a good investment.
• Indicators will skew management decisions in an unhelpful manner only if the indicators are irrelevant to the longer-term goals of the project (Outcomes or Impacts, depending on which agency’s jargon is involved).
• The process of developing indicators is itself a valuable part of the design process, helping - or forcing - participants to be clear with each other about what they mean when they talk about results.
• The arguments against bad indicators skewing behaviour are an argument for spending time to develop good indicators.

Measurement and verification

Finally, there is a third short and simple paper, produced separately at roughly the same time, that addresses these issues.  Rick Davies responded to Chris Whitehouse’ Ants and the Cockroach, with a one and a half-page commentary in September 2009.   His primary points, as I see them, are:

• No single indicator is ever likely to capture the process of change, but that multiple indicators might contribute differently to our understanding.
• Obsession with measurement is a problem - but while objectively verifiable indicators do not necessarily have to be measureable, they must be verifiable.
• Control groups are not the only means of attributing results to projects.  Most large projects have enough internal variation that with a little thought differences in results can be compared to differences in the way assistance was rendered.

Conclusions about indicators, results logic and assumptions

I agree with Thomas Winderl, and Rick Davies that indicator problems can usually be solved - if enough time and attention are paid to them.   


But for me the most interesting part of Rick Davies’ response is in how he deals with the horizontal and vertical logic of a project.  He notes that within the context of the Logical Framework, indicators are usually critiqued in terms of their external validity: do they measure what they are supposed to measure?


But, he adds, the more important question is whether the whole change process is actually clear: Do inputs, assumptions, completed activities, results and indicators hang together in a reasonable way?  Questioning this internal validity of project design is more important, and deserves more time than it gets now.  “We need more attention to theory, and perhaps a little less obsessing about measurement…. And a verified theory holds the potential of replicating successful processes of change elsewhere.”


In my experience, the serious examination of assumptions:  taking the time to clarify -- and then to monitor --assumptions about the development problem, assumptions about our theory of development, and assumptions about the working situation necessary for a reasonable chance of success, remains in practice one of the great unexplored areas of project implementation in a results-based management context.   If few resources are spent on developing and monitoring indicators, even fewer are allocated to the most fundamental of all issues in project and programme design:  examination of our assumptions.


Without serious attention to, and examination of  the multiple dimensions of the assumptions we are working with over the lifetime of a project, we are unlikely to learn lessons we can use in future practice, we are unlikely to identify underlying misunderstandings that can undermine our current work, and we are unlikely to spend public money in a responsible and effective manner.


The bottom line: None of these three very brief articles will walk you through the sometimes complex process of finding a good indicator, but they do point out the main arguments that have arisen over indicators, and which have been expressed in often more complex articles or books over the past 15 years.


Further reading on indicators and results


If this was too simple for you, there are more complex, yet interesting assessments of indicators and of alternative approaches to reporting and evaluation of results:

• Education for All, The Quality Imperative and the Problem of Pedagogy  by Robin Alexander examines in detail the failings of current commonly accepted indicators for education results.

• Why We Will Never Learn: A Political Economy of Aid Effectiveness,  by James Morton critiques major approaches to evaluation, and the disincentives for genuinely useful evaluations. 

• The Network of Networks on Impact Evaluation has a number of publications on how, if  you want, to conduct impact assessments.

_____________________________________________________________

GREG ARMSTRONG

Greg Armstrong is a Results-Based Management specialist who focuses on the use of clear language in RBM training, and in the creation of usable planning, monitoring and reporting frameworks.  For links to more Results-Based Management Handbooks and Guides, to to the RBM Training website.