google-site-verification: googlefccb558c724d02c7.html

Translate

Sunday, January 03, 2010

The LFA Debate: How Implementing Agencies View the Logical Framework

-- Greg Armstrong--


SIDA's LFA Papers:

2. The Use and Abuse of the Logical Framework Approach


This document reviews the many problems NGOs encounter in using the Logical Framework Approach.  

Level of difficulty:  Moderate to complex
Primarily useful for: Agency Planning and Evaluation groups, monitoring and evaluation specialists
Length:  28 pages
Most useful section:  Discussion of the LFA as a conceptual framework (p. 12-14)
Limitations:  Lacks a snappy concluding summary, and the findings are indicative, given the limited sample.


The LFA debate 


The Logical Framework is often associated with results-based management, and even when agencies dispense with the formal Logical Framework matrix, they almost always maintain some variation, such as a Performance Measurement Framework, as part of the planning and evaluation process.  For the past ten years, there has been an active debate on whether the Logical Framework Approach, and the associated matrices are helpful or harmful to understanding and improving development interventions, and much of this debate can be found in the Monitoring and Evaluation News archives and in academic research.  

There are three documents produced for one donor agency --- the Swedish International Development Agency (SIDA) between 2004-2006, however, that seem in part to reflect  something of the debate over whether - or how - to use the Logical Framework Approach in the design and management of development interventions.  

The first paper I reviewed, on December 27, 2009 was a summary of the theory behind the LFA, originally published in 2004. The next post will review a SIDA paper, published in 2006, that proposes the application of appreciative inquiry to the LFA.  

This post reviews The Use and Abuse of the Logical Framework Approach, by Oliver Bakewell and Anne Garbutt, of the organization INTRAC (The International NGO Training and Research Centre) in the U.K. Although the document itself, available from MandE and from INTRAC itself, has the SIDA logo on it, interestingly enough this document is not available directly from SIDA, and is not listed as one of the LFA-related documents available on the SIDA publications site. 


Who the Report is for


This report is not intended as a guide to practice.  It is a review of how different organizations, donors and NGOs, view the use of the LFA.   While many of the documents on the INTRAC website are oriented to practitioners, this one will probably not be of practical use to field officers, although it could inform discussion within aid agencies about whether and how to use or adapt the LFA.  


The report builds on both a review of literature (14 documents produced between 1990 and 2003), a survey of 18 agencies, and interviews with a selection of key informants in those agencies.  Officials of three donors are included as sources here - DfID, CIDA and FAO, along with those from 15 NGOs and consulting agencies.



How the LFA is used by Implementing Agencies



 The Logical Framework Approach in general is intended to tie together consultation with stakeholders, problem identification, assumptions, results definition and indicator development.
  • Some of the 15 implementing agencies responding to this study, reported using the Logical Framework matrix, but without going through the consultative process.
  • Others reported using the general approach, including consultation with stakeholders, and working through the relationship between interventions and results, but without using the matrix.
In theory, the whole process is supposed to relate not just to planning a project, but to managing inputs and activities for results, reporting on results, to monitoring, and to evaluation.  Yet, in reality, the authors note, "...in most cases the LFA is only explicitly used at the planning stage."



Problems with the Logical Framework Approach





Problems with participation and ownership




There are a couple of definite problems related to participation.  The report makes the point in several places that really incorporating genuine stakeholder participation in the Logical Framework process can be so time-consuming that many organizations minimise or abandon it.  

"It seems likely," this report notes, "that the individual's attitude to the LFA is likely to be closely related to how useful it is for their work rather than the nature of the organisation." 
But, the report suggests, a potential downside of genuine early participation in design of a project, is that ownership of the results, indicators and procedures becomes so strong, that nobody wants to make adjustments that are almost always necessary during project implementation.
"Some NGOs which had invested the time and effort in participatory planning reported finding that the resultant logical framework was such a valuable artefact, representing so many hours of negotiation to reach consensus, that it became very difficult to contemplate making further revisions as the project continued."


Strategies for coping with the LFA




 The study found three primary roles reported by implementing agencies, primarily NGOs, in using the LFA with their own stakeholders (smaller agencies or implementing groups)
  • Facilitator: NGOs sometimes work as facilitators of the process, helping their own stakeholders work through problem identification, results and indicator development, but this requires a huge amount of time.
  • Translator: NGOs sometimes serve as translators of existing planning processes, and turn that into different variations of the LFA depending on the varying formats required by different donors. 
  • Buffer: Sometimes implementing agencies simply act as a buffer between recipients and donors, selling the project with the LFA, which is later ignored during implementation.
The problems with implementing agencies acting as translators of the stakeholders' interests or as buffers between the stakeholders and donors are:
a)  Donors are divorced from the process, reducing their understanding of both the      programme and the organizations involved; and
b) The process of translating stakeholders' own analyses into a donor-acceptable Logical Framework matrix can be very complex.  "The process of logframe construction" one northern NGO respondent is quoted as saying in the report, "appears difficult for many international staff, even those with PhDs."


The LFA's adaptability - a myth


 The adaptive nature of the LFA - that, in theory, it can be changed as time and circumstances require - is a myth, according to this report.
"A major problem is the requirement in the LFA to work out the programme logic including identifying indicators from the outset, and the tendency, in practice, for that logic to become fixed in the matrix. In theory, the logical framework can be revised through the programme cycle and changes made, at least to the output level. In practice, this rarely happens....The rhetoric of flexibility and learning which is suggested by the theoretical application of the LFA does not work out in practice."



The LFA in Monitoring and Evaluation



Another significant criticism of the Logical Framework Approach reflected in this document is that donors often have little involvement in the participatory development of the original Logical Framework matrix with stakeholders. Consequently, they often have little understanding of the thinking behind the process, but later insist on using it as the basis for evaluations.


"They then take the resultant logical framework", the report says, "to evaluate a project or programme and ask (usually external) evaluators to use the indicators as the benchmarks for assessing the work." 


The LFA as a conceptual framework


The strength of the LFA is almost universally accepted as being its requirement for systematic thought about the relationship of activities to problems and results. On the other hand, the report concludes, it is too linear in its logic, leaving little room for unintended consequences, or the complexity of factors that may lead to results. 


Assumptions and Risk in the LFA


One very short, but important element of the report relates to assumptions:
"...one respondent argued that the management of risk and coping with the unexpected is critical for the success (or failure) of most development initiatives, and the risks and assumptions column is therefore the most important part of the logical framework matrix. However, it is usually the part taken the least seriously as it is the last column - more time is spent on outcomes and indicators. 'Risks are almost always poorly analysed and just put in for completeness' sake."


Conclusions about the LFA

 

The major strength of the LFA, seems to be, in this report, that it forces us to think through the logic of what we are doing, the relationship between the problem, activities and results.  But it is easy, the donors and NGOs seem to agree, to get locked into unrealistic expectations and assumptions.
"A simplistic characterisation of the prevailing attitudes to the LFA runs as follows: donors insist on it, while NGOs use it under sufferance. All recognise that it has many weaknesses, but there is a common view that despite these weaknesses, it is the best of a bad bunch of options available for planning and monitoring development work. Hence it carries on being widely used against all objections."
As an alternative, the report suggests:
"Rather than assessing whether we are delivering activities and outcomes in accordance with our grand theory set out in the logical framework, we should assess the theory. We may start with a set of expected activities and results, but as events unfold, the project or programme needs to respond, change its theory and revise its expectations. The emphasis should be on setting up systems for monitoring the impacts of our work, changing directions accordingly and monitoring those changes in direction. We need to constantly think how what we do relates to the goal, but we also have to hold that theory loosely and be prepared to learn and change."
The report suggests that implementing agencies should not be tied to the originally defined Outputs (usually, in donor-speak, the completion of activities) but should be free to make adaptations to reach the longer-term agreed upon results.  This would be the practical manifestation of genuinely "results-based" management. 



Learning Lessons from the Logical Framework Approach




The real issue here, the report suggests in its conclusion, is that lessons should be learned from development interventions, and they should be heeded both by those implementing projects and those planning and funding them.
 "Rather than criticising NGOs that carry out activities which then fail to contribute to the goal - i.e. that do not conform to the initial model - their sanctions should be reserved for those who fail to learn from this experience and carry on regardless."


Greg Armstrong's Comments on 9 Main LFA Issues:



1. The Logical Framework Matrix without Participation


Many organizations turn the use of the LFA from an "approach" which involves extensive participation, into mere Logical Framework analysis, which can be done, although without much beneficial effect, by an individual or small group. 

When the Logical Framework Approach degenerates to this level, when it becomes simply an individual or small group filling in the boxes in the matrix, the utility of the approach, for management, in my experience, diminishes to the point of nonexistence. The Logical Framework matrix then becomes a convenient tool for a donor agency to use as it summarizes and sells its projects to decision-makers, but without the grounded discussion that implementing agencies, field workers and aid recipients can bring to the process, it is likely to leave a development intervention that in practice will bear little relationship to the design, or the problems that the project hoped to address. 


2. Ownership and utility of the LFA


Among the organizations I have worked with it is, indeed, those which have found a way to use RBM in general, and particularly the LFA, to clarify their own thinking, which have used it most consistently, and most successfully. Those organizations that have used the LFA because they have been forced to do so,  rarely internalize it, or sustain the process. 

These would be the organizations this report describes as those that "... will prepare logical frameworks and jump through various LFA hoops when it is necessary to satisfy donors." 


3. Participation and delays in the development of the LFA - the role of donor agencies



It is not uncommon for the participatory component of project design within a donor agency to take up, perhaps, 4-5 months of time, and some donors shrink at that prospect, fearing design delays. But this is a total fantasy -- the delays, when they occur, are rarely attributable to genuine stakeholder participation, but instead to the time required within the donor agency to manipulate the data coming from the consultative process, to cope with the donor's internal political wrangling over shifting priorities, or the demands of changing development fashion. 

These bureaucratic issues can delay project development for up to three years. By that time, of course, the initial consultations among stakeholders may well be irrelevant, the original problem may have changed, and the design can easily have become outdated. 
So, pinning the blame for project delays on solid participatory needs assessment, results definition, and indicator development, seems to me to be a feeble attempt by donors to evade responsibility for delays. 


4. Complexity and Jargon in the Logical Framework process



This report points out, quite correctly, that thinking through the Logical Framework process can be complex, and the authors quote one source as saying the process is difficult too for international staff, “even those with PhDs”.

Working with NGOs, government organizations, private sector implementing agencies and universities, what I have seen is that it is precisely the people with PhDs, especially those working in universities, who have the biggest problems in working through the process. 

Getting a doctorate often involves the acquisition of sector-specific thinking patterns and jargon, something many of us know from personal experience. But I am convinced that the major reason the Logical Framework Approach and RBM in general are viewed as difficult is because the language used in these processes is obscure, and for most field workers unrelated to the reality of what occurs on the ground, in development projects. 

A simplified approach can overcome a lot of barriers to effective use of the Logical Framework Approach and to results-based management in general, and while this has proved to be very successful with NGOs and government agencies, the greatest resistance to using simple, plain language in the process is often met when working with universities. 



5. Is the Logical Framework Adaptable?


In my experience it is not universally true that the Logical Framework is too rigid to be adapted. I have seen at least four projects over the past five years, where indicators and sometimes results statements were changed mid-way through a project, after some sophisticated discussion among project stakeholders and the donor. These were all on projects worth between $5 million - $10 million, in fields as diverse as parliamentary development, environmental management and economic policy capacity development. 


6. Using the LFA for Evaluation: Donors' short attention span


It is definitely true, as this report suggests, that in many donor agencies there is often little institutionalized memory of the discussion process underlying development of the Logical Framework for individual programmes. 

Sometimes the donor representatives do conscientiously participate in these early project design discussions with stakeholders. But even when this occurs, delays in project approval, and the shifting of the original donor staff to new programme areas, often mean that with new staff, the donor agency does not have any continuous or embedded memory of the thinking processes underlying results statements, discussion of assumptions, indicators, or risks. 

Implementing agencies, however, usually do have this memory. For the implementing agency - providing it is not just another donor agency (as when bilateral donors fund projects by UN agencies) the senior staff often remain with the project, and have adapted to changing circumstances with new strategies, and perhaps new ideas about what the results should be. 


7. Baseline data: The weak foundation of many indicators 


On the other hand, it seems to me that the claim of rigidity in applying indicators is actually the reverse of reality in many cases. 

It is unfortunately rare for donors to actually use the indicators in the Logical Framework as the foundation for an evaluation.  If  the indicator development discussions are done right, if realistic indicators are chosen by stakeholders and permitted to be adapted over time -- as I have seen done on several occasions -- then using the agreed-upon results and indicators as a starting point for the evaluation is reasonable. But there are two important issues here:
  • Baseline data is often never collected at all on many large projects.  This undermines the utility of developing indicators, which could, if they were tracked, help us learn if we have a reasonably defined result, and help us test the assumptions in our results chains.
  • Many donors simply don't appear to be interested in whether there is baseline data, or not.  Many don't seem to notice if any baseline data has been collected, or don't really care, if they do notice. Often the donor agencies just don't want to rock the contractual boat, or to be perceived as predatory, expecting any hard-nosed accountability.   
I cannot remember the last time I saw a donor agency tell implementers that funding would be affected if baseline data were not collected.   It is reported that DfID may be moving to require baseline data early in project implementation, and while there is some fear that it will make the process of evaluation and planning rigid, I think it is a good idea.  

While rigidity is obviously not desirable, it seems to me that the greater risk is that public funds will be spent without any clearly articulated idea of why they are being spent, and whether results are likely to be realised.  The threat, eventually, without some genuine accountability for results, will be to the perceived legitimacy of aid programmes, by those footing the bill.

The casual attitude of some donors towards baseline data, however, is consistent with one of the key points of this study -- that the Logical Framework Approach, (and by extension, it seems to me, results-based management) -- is being used by donors primarily for planning, but not really for management. 



8. The LFA and attribution of results




It is interesting that it is the implementing agencies in many cases that seem to buy into the "cause and effect" theory of activities leading to results -- at least if the results appear to be achieved.  It is donors, increasingly, that are questioning the attribution of results solely to funded development interventions.  In Canada, for example, for several years, the Treasury Board, the Auditor General and more recently CIDA have cautioned against making exorbitant or unrealistic claims for the successes of development interventions -- and have called for a more subtle and nuanced discussion of how projects and programmes contribute to long-term results. 


9. Unexamined Assumptions in the LFA




The quote in the report to the effect that  “…the risks and assumptions column is therefore the most important part of the logical framework matrix.... However, it is usually the part taken the least seriously as it is the last column - more time is spent on outcomes and indicators, “ is perceptive.  

It is after all, our assumptions that we are testing when we undertake development projects or programmes:


  • Do we share assumptions about what the problem is?
  • Do we share assumptions about cause and effect in development interventions - what works and what does not?
  • Do we understand (and share) the assumptions we are making about the behaviour of other actors, if our interventions are likely to contribute to results?
  • Are we prepared, and permitted by the donor, to act to adapt or terminate the project or programme, if we learn that our assumptions about the problem, our assumptions about the logic of our intervention, or our assumptions about the situation and other actors are wrong?
Many donors do deal, superficially, with risk -- but often it is treated as an afterthought, in terms of whether an occurring risk should trigger management changes.

Assumptions are given even less attention -- rarely detailed or treated seriously as a factor affecting an intervention. In some cases, where RBM frameworks change, the discussion of assumptions drops out of the discussion completely.

If, as the report suggests, the major strength of the Logical Framework Approach is that it "...does force people to think through their theory of change", then clarifying the assumptions beneath these espoused theories, and revisiting them as theories in use  as the project advances, should be an important part of the life of a development intervention.



 Limitations to the report

There are three limitations to the utility of this report:

1. Some organizations, such as UNDP, play roles at different times as both the donor and the implementing agency, with very mixed results.  It would have been interesting to get the UNDP view on the questions the authors were exploring.

2. This paper does not provide a snappy summary of the problems or potential advantages of the LFA.  Readers who want that, however, can get it from several documents on the MandE website.

3. It remains to be seen, based on the sample size, if the findings are generalizable, or relevant to issues such as programme-based approaches, but the authors do not make any claim for this.

The bottom line:  This is not a guide to practice; it is a report on how some organizations use the LFA. This paper is not arguing for some radical alternative to the Logical Framework Approach, but it does suggest that more flexible approach to its use would be useful.  It could be useful as background to the discussion on the appropriate uses of results-based management for development, primarily for planners and aid theorists.



Further reading:








 
Read the latest posts