-- Greg Armstrong--
SIDA's LFA Papers:
Primarily useful for: Agency Planning and Evaluation groups, monitoring and evaluation specialists
Length: 28 pages
Most useful section: Discussion of the LFA as a conceptual framework (p. 12-14)
Limitations: Lacks a snappy concluding summary, and the findings are indicative, given the limited sample.
[Edited to update links, June 2018]
SIDA's LFA Papers:
2. The Use and Abuse of the Logical Framework Approach
This document reviews the many problems NGOs encounter in using the Logical Framework Approach.
Level of difficulty: Moderate to complexThis document reviews the many problems NGOs encounter in using the Logical Framework Approach.
Primarily useful for: Agency Planning and Evaluation groups, monitoring and evaluation specialists
Length: 28 pages
Most useful section: Discussion of the LFA as a conceptual framework (p. 12-14)
Limitations: Lacks a snappy concluding summary, and the findings are indicative, given the limited sample.
[Edited to update links, June 2018]
The LFA debate
The Logical Framework is often associated with results-based management, and even when agencies dispense with the formal Logical Framework matrix, they almost always maintain some variation, such as a Performance Measurement Framework, as part of the planning and evaluation process. For the past ten years, there has been an active debate on whether the Logical Framework Approach, and the associated matrices are helpful or harmful to understanding and improving development interventions, and much of this debate can be found in the Monitoring and Evaluation News archives and in academic research.
There are three documents produced for one donor agency --- the Swedish International Development Agency (SIDA) between 2004-2006, however, that seem in part to reflect something of the debate over whether - or how - to use the Logical Framework Approach in the design and management of development interventions.
The first paper I reviewed, on December 27, 2009 was a summary of the theory behind the LFA, originally published in 2004. The next post will review a SIDA paper, published in 2006, that proposes the application of appreciative inquiry to the LFA.
This post reviews The Use and Abuse of the Logical Framework Approach, (PDF) by Oliver Bakewell and Anne Garbutt, of the organization INTRAC (The International NGO Training and Research Centre) in the U.K. Although the document itself, once, but no longer, available from both MandE and from INTRAC itself, has the SIDA logo on it, interestingly enough this document is not available directly from SIDA, and is not listed as one of the LFA-related documents available on the SIDA publications site.
Who the Report is for
This report is not intended as a guide to practice. It is a review of how different organizations, donors and NGOs, view the use of the LFA. While many of the documents on the INTRAC website are oriented to practitioners, this one will probably not be of practical use to field officers, although it could inform discussion within aid agencies about whether and how to use or adapt the LFA.
The report builds on both a review of literature (14 documents produced between 1990 and 2003), a survey of 18 agencies, and interviews with a selection of key informants in those agencies. Officials of three donors are included as sources here - DfID, CIDA and FAO, along with those from 15 NGOs and consulting agencies.
The Logical Framework Approach in general is intended to tie together consultation with stakeholders, problem identification, assumptions, results definition and indicator development.
There are a couple of definite problems related to participation. The report makes the point in several places that really incorporating genuine stakeholder participation in the Logical Framework process can be so time-consuming that many organizations minimise or abandon it.
The study found three primary roles reported by implementing agencies, primarily NGOs, in using the LFA with their own stakeholders (smaller agencies or implementing groups)
The adaptive nature of the LFA - that, in theory, it can be changed as time and circumstances require - is a myth, according to this report.
Another significant criticism of the Logical Framework Approach reflected in this document is that donors often have little involvement in the participatory development of the original Logical Framework matrix with stakeholders. Consequently, they often have little understanding of the thinking behind the process, but later insist on using it as the basis for evaluations.
"They then take the resultant logical framework", the report says, "to evaluate a project or programme and ask (usually external) evaluators to use the indicators as the benchmarks for assessing the work."
This report points out, quite correctly, that thinking through the Logical Framework process can be complex, and the authors quote one source as saying the process is difficult too for international staff, “even those with PhDs”.
While rigidity is obviously not desirable, it seems to me that the greater risk is that public funds will be spent without any clearly articulated idea of why they are being spent, and whether results are likely to be realised. The threat, eventually, without some genuine accountability for results, will be to the perceived legitimacy of aid programmes, by those footing the bill.
The casual attitude of some donors towards baseline data, however, is consistent with one of the key points of this study -- that the Logical Framework Approach, (and by extension, it seems to me, results-based management) -- is being used by donors primarily for planning, but not really for management.
It is interesting that it is the implementing agencies in many cases that seem to buy into the "cause and effect" theory of activities leading to results -- at least if the results appear to be achieved. It is donors, increasingly, that are questioning the attribution of results solely to funded development interventions. In Canada, for example, for several years, the Treasury Board, the Auditor General and more recently CIDA have cautioned against making exorbitant or unrealistic claims for the successes of development interventions -- and have called for a more subtle and nuanced discussion of how projects and programmes contribute to long-term results.
The quote in the report to the effect that “…the risks and assumptions column is therefore the most important part of the logical framework matrix.... However, it is usually the part taken the least seriously as it is the last column - more time is spent on outcomes and indicators, “ is perceptive.
It is after all, our assumptions that we are testing when we undertake development projects or programmes:
Assumptions are given even less attention -- rarely detailed or treated seriously as a factor affecting an intervention. In some cases, where RBM frameworks change, the discussion of assumptions drops out of the discussion completely.
If, as the report suggests, the major strength of the Logical Framework Approach is that it "...does force people to think through their theory of change", then clarifying the assumptions beneath these espoused theories, and revisiting them as theories in use as the project advances, should be an important part of the life of a development intervention.
Limitations to the report
There are three limitations to the utility of this report:
1. Some organizations, such as UNDP, play roles at different times as both the donor and the implementing agency, with very mixed results. It would have been interesting to get the UNDP view on the questions the authors were exploring.
2. This paper does not provide a snappy summary of the problems or potential advantages of the LFA. Readers who want that, however, can get it from several documents on the MandE website.
3. It remains to be seen, based on the sample size, if the findings are generalizable, or relevant to issues such as programme-based approaches, but the authors do not make any claim for this.
The bottom line: This is not a guide to practice; it is a report on how some organizations use the LFA. This paper is not arguing for some radical alternative to the Logical Framework Approach, but it does suggest that more flexible approach to its use would be useful. It could be useful as background to the discussion on the appropriate uses of results-based management for development, primarily for planners and aid theorists.
GREG ARMSTRONG
Greg Armstrong is a Results-Based Management specialist who focuses on the use of clear language in RBM training, and in the creation of usable planning, monitoring and reporting frameworks. For links to more Results-Based Management Handbooks and Guides, go to the RBM Training website.
The Logical Framework is often associated with results-based management, and even when agencies dispense with the formal Logical Framework matrix, they almost always maintain some variation, such as a Performance Measurement Framework, as part of the planning and evaluation process. For the past ten years, there has been an active debate on whether the Logical Framework Approach, and the associated matrices are helpful or harmful to understanding and improving development interventions, and much of this debate can be found in the Monitoring and Evaluation News archives and in academic research.
There are three documents produced for one donor agency --- the Swedish International Development Agency (SIDA) between 2004-2006, however, that seem in part to reflect something of the debate over whether - or how - to use the Logical Framework Approach in the design and management of development interventions.
The first paper I reviewed, on December 27, 2009 was a summary of the theory behind the LFA, originally published in 2004. The next post will review a SIDA paper, published in 2006, that proposes the application of appreciative inquiry to the LFA.
This post reviews The Use and Abuse of the Logical Framework Approach, (PDF) by Oliver Bakewell and Anne Garbutt, of the organization INTRAC (The International NGO Training and Research Centre) in the U.K. Although the document itself, once, but no longer, available from both MandE and from INTRAC itself, has the SIDA logo on it, interestingly enough this document is not available directly from SIDA, and is not listed as one of the LFA-related documents available on the SIDA publications site.
Who the Report is for
This report is not intended as a guide to practice. It is a review of how different organizations, donors and NGOs, view the use of the LFA. While many of the documents on the INTRAC website are oriented to practitioners, this one will probably not be of practical use to field officers, although it could inform discussion within aid agencies about whether and how to use or adapt the LFA.
The report builds on both a review of literature (14 documents produced between 1990 and 2003), a survey of 18 agencies, and interviews with a selection of key informants in those agencies. Officials of three donors are included as sources here - DfID, CIDA and FAO, along with those from 15 NGOs and consulting agencies.
How the LFA is used by Implementing Agencies
The Logical Framework Approach in general is intended to tie together consultation with stakeholders, problem identification, assumptions, results definition and indicator development.
- Some of the 15 implementing agencies responding to this study, reported using the Logical Framework matrix, but without going through the consultative process.
- Others reported using the general approach, including consultation with stakeholders, and working through the relationship between interventions and results, but without using the matrix.
Problems with the Logical Framework Approach
Problems with participation and ownership
There are a couple of definite problems related to participation. The report makes the point in several places that really incorporating genuine stakeholder participation in the Logical Framework process can be so time-consuming that many organizations minimise or abandon it.
"It seems likely," this report notes, "that the individual's attitude to the LFA is likely to be closely related to how useful it is for their work rather than the nature of the organisation."But, the report suggests, a potential downside of genuine early participation in design of a project, is that ownership of the results, indicators and procedures becomes so strong, that nobody wants to make adjustments that are almost always necessary during project implementation.
"Some NGOs which had invested the time and effort in participatory planning reported finding that the resultant logical framework was such a valuable artefact, representing so many hours of negotiation to reach consensus, that it became very difficult to contemplate making further revisions as the project continued."
Strategies for coping with the LFA
The study found three primary roles reported by implementing agencies, primarily NGOs, in using the LFA with their own stakeholders (smaller agencies or implementing groups)
- Facilitator: NGOs sometimes work as facilitators of the process, helping their own stakeholders work through problem identification, results and indicator development, but this requires a huge amount of time.
- Translator: NGOs sometimes serve as translators of existing planning processes, and turn that into different variations of the LFA depending on the varying formats required by different donors.
- Buffer: Sometimes implementing agencies simply act as a buffer between recipients and donors, selling the project with the LFA, which is later ignored during implementation.
The problems with implementing agencies acting as translators of the stakeholders' interests or as buffers between the stakeholders and donors are:
a) Donors are divorced from the process, reducing their understanding of both the programme and the organizations involved; and
b) The process of translating stakeholders' own analyses into a donor-acceptable Logical Framework matrix can be very complex. "The process of logframe construction" one northern NGO respondent is quoted as saying in the report, "appears difficult for many international staff, even those with PhDs."
The LFA's adaptability - a myth
The adaptive nature of the LFA - that, in theory, it can be changed as time and circumstances require - is a myth, according to this report.
"A major problem is the requirement in the LFA to work out the programme logic including identifying indicators from the outset, and the tendency, in practice, for that logic to become fixed in the matrix. In theory, the logical framework can be revised through the programme cycle and changes made, at least to the output level. In practice, this rarely happens....The rhetoric of flexibility and learning which is suggested by the theoretical application of the LFA does not work out in practice."
The LFA in Monitoring and Evaluation
Another significant criticism of the Logical Framework Approach reflected in this document is that donors often have little involvement in the participatory development of the original Logical Framework matrix with stakeholders. Consequently, they often have little understanding of the thinking behind the process, but later insist on using it as the basis for evaluations.
The LFA as a conceptual framework
The strength of the LFA is almost universally accepted as being its requirement for systematic thought about the relationship of activities to problems and results. On the other hand, the report concludes, it is too linear in its logic, leaving little room for unintended consequences, or the complexity of factors that may lead to results.
Assumptions and Risk in the LFA
One very short, but important element of the report relates to assumptions:
The major strength of the LFA, seems to be, in this report, that it forces us to think through the logic of what we are doing, the relationship between the problem, activities and results. But it is easy, the donors and NGOs seem to agree, to get locked into unrealistic expectations and assumptions.
The real issue here, the report suggests in its conclusion, is that lessons should be learned from development interventions, and they should be heeded both by those implementing projects and those planning and funding them.
"...one respondent argued that the management of risk and coping with the unexpected is critical for the success (or failure) of most development initiatives, and the risks and assumptions column is therefore the most important part of the logical framework matrix. However, it is usually the part taken the least seriously as it is the last column - more time is spent on outcomes and indicators. 'Risks are almost always poorly analysed and just put in for completeness' sake."
Conclusions about the LFA
The major strength of the LFA, seems to be, in this report, that it forces us to think through the logic of what we are doing, the relationship between the problem, activities and results. But it is easy, the donors and NGOs seem to agree, to get locked into unrealistic expectations and assumptions.
"A simplistic characterisation of the prevailing attitudes to the LFA runs as follows: donors insist on it, while NGOs use it under sufferance. All recognise that it has many weaknesses, but there is a common view that despite these weaknesses, it is the best of a bad bunch of options available for planning and monitoring development work. Hence it carries on being widely used against all objections."As an alternative, the report suggests:
"Rather than assessing whether we are delivering activities and outcomes in accordance with our grand theory set out in the logical framework, we should assess the theory. We may start with a set of expected activities and results, but as events unfold, the project or programme needs to respond, change its theory and revise its expectations. The emphasis should be on setting up systems for monitoring the impacts of our work, changing directions accordingly and monitoring those changes in direction. We need to constantly think how what we do relates to the goal, but we also have to hold that theory loosely and be prepared to learn and change."The report suggests that implementing agencies should not be tied to the originally defined Outputs (usually, in donor-speak, the completion of activities) but should be free to make adaptations to reach the longer-term agreed upon results. This would be the practical manifestation of genuinely "results-based" management.
Learning Lessons from the Logical Framework Approach
The real issue here, the report suggests in its conclusion, is that lessons should be learned from development interventions, and they should be heeded both by those implementing projects and those planning and funding them.
"Rather than criticising NGOs that carry out activities which then fail to contribute to the goal - i.e. that do not conform to the initial model - their sanctions should be reserved for those who fail to learn from this experience and carry on regardless."
Greg Armstrong's Comments on 9 Main LFA Issues:
1. The Logical Framework Matrix without Participation
Many organizations turn the use of the LFA from an "approach" which involves extensive participation, into mere Logical Framework analysis, which can be done, although without much beneficial effect, by an individual or small group.
When the Logical Framework Approach degenerates to this level, when it becomes simply an individual or small group filling in the boxes in the matrix, the utility of the approach, for management, in my experience, diminishes to the point of nonexistence. The Logical Framework matrix then becomes a convenient tool for a donor agency to use as it summarizes and sells its projects to decision-makers, but without the grounded discussion that implementing agencies, field workers and aid recipients can bring to the process, it is likely to leave a development intervention that in practice will bear little relationship to the design, or the problems that the project hoped to address.
2. Ownership and utility of the LFA
Among the organizations I have worked with it is, indeed, those which have found a way to use RBM in general, and particularly the LFA, to clarify their own thinking, which have used it most consistently, and most successfully. Those organizations that have used the LFA because they have been forced to do so, rarely internalize it, or sustain the process.
These would be the organizations this report describes as those that "... will prepare logical frameworks and jump through various LFA hoops when it is necessary to satisfy donors."
3. Participation and delays in the development of the LFA - the role of donor agencies
It is not uncommon for the participatory component of project design within a donor agency to take up, perhaps, 4-5 months of time, and some donors shrink at that prospect, fearing design delays. But this is a total fantasy -- the delays, when they occur, are rarely attributable to genuine stakeholder participation, but instead to the time required within the donor agency to manipulate the data coming from the consultative process, to cope with the donor's internal political wrangling over shifting priorities, or the demands of changing development fashion.
These bureaucratic issues can delay project development for up to three years. By that time, of course, the initial consultations among stakeholders may well be irrelevant, the original problem may have changed, and the design can easily have become outdated.
So, pinning the blame for project delays on solid participatory needs assessment, results definition, and indicator development, seems to me to be a feeble attempt by donors to evade responsibility for delays.
4. Complexity and Jargon in the Logical Framework process
This report points out, quite correctly, that thinking through the Logical Framework process can be complex, and the authors quote one source as saying the process is difficult too for international staff, “even those with PhDs”.
Working with NGOs, government organizations, private sector implementing agencies and universities, what I have seen is that it is precisely the people with PhDs, especially those working in universities, who have the biggest problems in working through the process.
Getting a doctorate often involves the acquisition of sector-specific thinking patterns and jargon, something many of us know from personal experience. But I am convinced that the major reason the Logical Framework Approach and RBM in general are viewed as difficult is because the language used in these processes is obscure, and for most field workers unrelated to the reality of what occurs on the ground, in development projects.
A simplified approach can overcome a lot of barriers to effective use of the Logical Framework Approach and to results-based management in general, and while this has proved to be very successful with NGOs and government agencies, the greatest resistance to using simple, plain language in the process is often met when working with universities.
5. Is the Logical Framework Adaptable?
In my experience it is not universally true that the Logical Framework is too rigid to be adapted. I have seen at least four projects over the past five years, where indicators and sometimes results statements were changed mid-way through a project, after some sophisticated discussion among project stakeholders and the donor. These were all on projects worth between $5 million - $10 million, in fields as diverse as parliamentary development, environmental management and economic policy capacity development.
6. Using the LFA for Evaluation: Donors' short attention span
It is definitely true, as this report suggests, that in many donor agencies there is often little institutionalized memory of the discussion process underlying development of the Logical Framework for individual programmes.
Sometimes the donor representatives do conscientiously participate in these early project design discussions with stakeholders. But even when this occurs, delays in project approval, and the shifting of the original donor staff to new programme areas, often mean that with new staff, the donor agency does not have any continuous or embedded memory of the thinking processes underlying results statements, discussion of assumptions, indicators, or risks.
Implementing agencies, however, usually do have this memory. For the implementing agency - providing it is not just another donor agency (as when bilateral donors fund projects by UN agencies) the senior staff often remain with the project, and have adapted to changing circumstances with new strategies, and perhaps new ideas about what the results should be.
7. Baseline data: The weak foundation of many indicators
On the other hand, it seems to me that the claim of rigidity in applying indicators is actually the reverse of reality in many cases.
It is unfortunately rare for donors to actually use the indicators in the Logical Framework as the foundation for an evaluation. If the indicator development discussions are done right, if realistic indicators are chosen by stakeholders and permitted to be adapted over time -- as I have seen done on several occasions -- then using the agreed-upon results and indicators as a starting point for the evaluation is reasonable. But there are two important issues here:
- Baseline data is often never collected at all on many large projects. This undermines the utility of developing indicators, which could, if they were tracked, help us learn if we have a reasonably defined result, and help us test the assumptions in our results chains.
- Many donors simply don't appear to be interested in whether there is baseline data, or not. Many don't seem to notice if any baseline data has been collected, or don't really care, if they do notice. Often the donor agencies just don't want to rock the contractual boat, or to be perceived as predatory, expecting any hard-nosed accountability.
While rigidity is obviously not desirable, it seems to me that the greater risk is that public funds will be spent without any clearly articulated idea of why they are being spent, and whether results are likely to be realised. The threat, eventually, without some genuine accountability for results, will be to the perceived legitimacy of aid programmes, by those footing the bill.
The casual attitude of some donors towards baseline data, however, is consistent with one of the key points of this study -- that the Logical Framework Approach, (and by extension, it seems to me, results-based management) -- is being used by donors primarily for planning, but not really for management.
8. The LFA and attribution of results
It is interesting that it is the implementing agencies in many cases that seem to buy into the "cause and effect" theory of activities leading to results -- at least if the results appear to be achieved. It is donors, increasingly, that are questioning the attribution of results solely to funded development interventions. In Canada, for example, for several years, the Treasury Board, the Auditor General and more recently CIDA have cautioned against making exorbitant or unrealistic claims for the successes of development interventions -- and have called for a more subtle and nuanced discussion of how projects and programmes contribute to long-term results.
9. Unexamined Assumptions in the LFA
The quote in the report to the effect that “…the risks and assumptions column is therefore the most important part of the logical framework matrix.... However, it is usually the part taken the least seriously as it is the last column - more time is spent on outcomes and indicators, “ is perceptive.
It is after all, our assumptions that we are testing when we undertake development projects or programmes:
- Do we share assumptions about what the problem is?
- Do we share assumptions about cause and effect in development interventions - what works and what does not?
- Do we understand (and share) the assumptions we are making about the behaviour of other actors, if our interventions are likely to contribute to results?
- Are we prepared, and permitted by the donor, to act to adapt or terminate the project or programme, if we learn that our assumptions about the problem, our assumptions about the logic of our intervention, or our assumptions about the situation and other actors are wrong?
Assumptions are given even less attention -- rarely detailed or treated seriously as a factor affecting an intervention. In some cases, where RBM frameworks change, the discussion of assumptions drops out of the discussion completely.
If, as the report suggests, the major strength of the Logical Framework Approach is that it "...does force people to think through their theory of change", then clarifying the assumptions beneath these espoused theories, and revisiting them as theories in use as the project advances, should be an important part of the life of a development intervention.
Limitations to the report
There are three limitations to the utility of this report:
1. Some organizations, such as UNDP, play roles at different times as both the donor and the implementing agency, with very mixed results. It would have been interesting to get the UNDP view on the questions the authors were exploring.
2. This paper does not provide a snappy summary of the problems or potential advantages of the LFA. Readers who want that, however, can get it from several documents on the MandE website.
3. It remains to be seen, based on the sample size, if the findings are generalizable, or relevant to issues such as programme-based approaches, but the authors do not make any claim for this.
The bottom line: This is not a guide to practice; it is a report on how some organizations use the LFA. This paper is not arguing for some radical alternative to the Logical Framework Approach, but it does suggest that more flexible approach to its use would be useful. It could be useful as background to the discussion on the appropriate uses of results-based management for development, primarily for planners and aid theorists.
Further reading:
_____________________________________________________________- See the interesting discussion between Philip Dearden and Rick Davies between September 25 and October 10, 2009, on DfID's move toward more stringent baseline data.
- For those interested in an analysis of the conceptual issues involved in applying the LFA to health sector projects, see Reidar Dale's 2003 article (one of the references for the currently reviewed report) "The Logical Framework: An easy escape, a straitjacket, or a useful planning tool?"
