The dangers of using standard performance indicators in development projects

by Greg Armstrong

[Updated October 2018]

International aid agencies sometimes try to use standard performance indicators to make aggregation of agency-level results  easier, but there are real dangers in using these at the project level as Sara Holzapfel  explains in her detailed report The Role of Indicators in Development Cooperation

Level of Complexity:  Complex
Length:  236 pages (the full version, 38 pages for an earlier discussion paper)
Languages: English
Primarily Useful for: Agency-level RBM experts
Most Useful:  Comparisons of how aid agencies use standard indicators at different results levels - particularly Appendix A5, p. 205-233.
Limitations:  Realistically, there are no quick answers on how to aggregate project-level results without using standard indicators

Sara Holzapfel:  The role of indicators in development cooperation

Background

Many of us have encountered development projects where, in the attempt to demonstrate agency-wide accountability for results  the aid agency designing the project has moved away from project-specific results indicators which measure capacity development, policy change or changes in implementation, and attempted to use agency-level standard indicators at the project level.  

As Sara Holzapfel notes in her detailed assessment of how aid agencies use standard performance indicators, published by the: German Development Institute  this trend has arisen both from a concern with accountability and a real interest in whether development programmes contribute to agreed upon goals.

“Amid rising criticism of aid effectiveness coupled with tight budgets in many traditional donor countries at a time of economic crisis, donor agencies are under pressure to deliver more value for money and to provide evidence of the positive effects of development cooperation. In response to these pressures, more and more development agencies are adopting agency results frameworks for monitoring and managing their progress in pursuing their strategic objectives and for reporting on performance.”

Even where there is apparent, substantial progress on broad, long-term indicators, such as the millennium development goals  questions still remain about whether the indicators for these goals, themselves, can tell us anything about whether development interventions contribute effectively to long term results, whether the indicators describe progress against problems effectively  and as a 2014 report on MDG targets and indicators for human development and human rights published in the Journal of Human Development and Capabilities noted 

“The unintended consequences revealed in the Project cannot merely be ascribed to the goals and targets having been selected or implemented badly, as is sometimes claimed. They are more fundamental structural issues arising from the nature of quantification and the nested structure of goals, targets and indicators that the MDGs created.”

Standard and custom development indicators

Governance Assessments and Governance Indicators – The Governance of Forests Initiative Indicator Guidance Manual

By GregArmstrong

[Updated August 2019]

Don’t let the title mislead you - The Governance of Forests Initiative (GFI) Indicator Framework and the accompanying Guidance Manual published  in 2013 by the World Resources Institute are practical tools not just for those working on environmental governance, but for anyone developing indicators for the design, management or monitoring of governance  projects or programmes in any sector.

Level of Difficulty: Moderate to Complex
Length: 295 pages - or downloadable as 7 discrete files by component
Languages:  English
Primarily Useful For: Indicator workshop facilitators, project or programme managers
Most Useful:  Guidance on data collection for 596 indicators
Limitations: These are not cut and paste indicators.  The Guide provides examples of issues to be considered in data collection, but users will need to do their own work to adapt indicators to specific contexts



Background

Developing indicators for governance projects and programmes– whether democratic governance in general, or governance in specific sectors such as natural resources or the environment -  always requires careful discussion and debate among those designing and managing projects, and their partners and stakeholders.  .

There are usually very few shortcuts we can take in working through what potential indicators are at the same time both technically and politically valid, and practical enough to actually have collectable data.  In my experience on both environmental governance and democratic governance projects, it is necessary often to start from scratch on each unique project, clarifying assumptions about what the results mean to different groups, and what is likely to be practical in terms of data collection.  For projects on transparency, democratic governance, and accountability it is often counterproductive to simply adopt standard global indicators used to compare country performance on governance, and try to use them to describe specific capacity development needs of individual governance projects.

Useful publications from the World Resources Institute 

I was therefore pleasantly surprised to discover, belatedly, the World Resources Institute and its publications, particularly those on The Governance of Forests Initiative.  The WRI states that while it ensures that its publications meet academic standards, “We also ensure that all of our publications are timely, fit for audience, and rooted in a strategic plan for achieving positive change in the world.”  Based on my reading of the indicator guidance manual, this is certainly fit for the audience of people I work with – people designing, managing , monitoring ,evaluating, and affected by governance projects – in forestry, natural resources management or any other sector.

Practical governance indicator guidance

There are dozens of reports and guides on governance issues and governance indicators on the WRI website, but after sorting through many of them, what I found the most useful for facilitators of indicator development workshops for people working in the field on governance projects in a wide range of sectors is the series of publications on indicators for assessing forest governance.  These were field tested over several years in Cameroon, Indonesia and Brazil, before publication.

For those wanting just an overview of the process of assessing governance, the 68-page Assessing Forest Governance: The Governance of Forests Initiative Indicator Framework might be of interest.  It is, however, somewhat mislabeled I think, on the publications download page  as a “full report”.

There are numerous alternatives for downloading the indicator framework and reports, a summary or individual chapters at the WRI website.

Governance of Forest Initiatives Indicator framework download alternatives
[click to enlarge

The 295-page Governance of Forests Initiative (GFI) Guidance Manual: A Guide to Using the GFI Indicator Framework , written by Lauren Williams, Jessica Breitfeller and  Celine Lim, while not as pretty as the framework itself, contains, as far as I could see, both the report, and much more detailed discussion of the practical issues involved in actually using the indicators, or adapting them to project or community specific contexts. It is this document which I think will be of greatest use to people who need not just to develop governance indicators, but to specify how they will be defined and how data will be collected.

This Guidance manual is divided into two parts – an introductory section on how to conduct governance assessments, and detailed guides to collecting data on 596 indicators in part 2.  

While I think the whole Guide is worth downloading as one coherent document, the website also,  for those with limited bandwidth, interest or patience, provides links to 7 sections with theme-specific indicators which can be downloaded separately. The rest of this post describes what is in those sections, and in the Guide to using the indicator framework.

How to conduct a governance assessment

Principles of good governance

 This 35-page introductory Part 1 of the guide discusses the principles of good governance for which it is proposing indicators, including transparency, participation, accountability, coordination and “capacity”, and four forestry themes with which they will be matched in indicator development.  

Matching Governance Themes and Indicators
[Click to enlarge}

Those themes are forest tenure, land use, forest management, forest revenues.  But the Guide also recognizes the need for indicators on cross-cutting governance institutions and cross-cutting governance issues of relevance to any sector.