In 17 years, about 1.6 trillion dollars spent on sustainable development; very few undergo post-project evaluation

In a candid interview, Jindra Cekan tells SixDegrees why most projects may not be sustainable as so few projects have ever been evaluated after ending and how project design and implementation can be flawed.

Interview/Jindra Cekan

Dr Jindra Cekan has used participatory methods for 30 years to connect with participants and policy makers, ranging from villagers to ministers in Africa, Central & Latin America and the Balkans. Their knowledge has informed monitoring and evaluation for strategic planning content, project design, organizational knowledge management and learning. She has also led assessments, baselines and final evaluations, and especially sustained and emerging impacts evaluations. She has a focus on monitoring and evaluation (M&E), food security/ livelihoods and gender. Dr Cekan founded Cekan Consulting LLC 15 years ago, serving international development clients in 28 countries, including government, foundations, non-profit and for-profit enterprises. She founded Valuing Voices at Cekan Consulting, the evaluation and advocacy network in 2013 for post-project sustainability evaluations.

Why did you launch Valuing Voices?

One of the reasons for founding Valuing Voices is how can I be a voice for the Asians, Latin Americans, Africans so rarely heard? I am American, but I am also Czech. The people whose interest I am trying to advocate for are our village participants, our non-profit and government partners in countries. I have found that so little of these projects is country led, so few of their voices heard in project planning or M&E, which I find beyond scandalous.

I worked in international development since 1988 when I first went to Nigeria, then I got my PhD and went to Mali. I worked subsequently in numerous countries like Kenya, Ethiopia, Peru, Honduras. My focus is on food security, livelihood and microenterprise and for many years, oversaw programmes in Catholic Relief Services. This included work for the American Red Cross and ICRC, overseeing Kosovo food relief during the refugee crisis, as later as a consultant for project design and evaluation. Five years ago, I was consulting to USAID for organizational development and M&E. I wondered about the sustainability of evaluation and at the very end of my assignment, I asked them whether anybody is funding post project evaluation as it seems it is done very rarely. “Do you know anyone working on it? This has been an interest of mine and wanted to know are we learning from going back to these now-closed projects.” The group which funds all the M&E at USAID said no one is working on post project evaluation since the last time USAID did this in the 1970s-1980s. I asked the reason-why not?  They said that the reason is because our responsibility is to monitor during the donor funded project, not after because that is the responsibility of the people whom we hand over to. I was surprised, to say the least. Later they informed me they would fund about a million and a half dollars of post project evaluation of education under a grant for Uganda, South Africa, Ghana and Namibia on sustainability. I was separately selected to lead the Uganda case.

In 2013, I started Valuing Voices researching whether it was possible that nobody returns, yet we talk about “sustainable development”. I felt sure I would find published research, and why not me? Somebody has to do it. Hence, I spent the first two years of Valuing Voices researching (in between paid consulting projects) looking for research that evaluated results after funding ended. I found a handful of such evaluations.

You have been raising concern that the international aid system is dysfunctional, and projects are non-sustainable-can you please elaborate on this?

I know that most projects are designed in donor offices based on the kinds of funding available and then implementers revise programming on the ground. I know that most monitoring and evaluation of results is done to report to donors, and while many participants are asked questions, rarely is such data or analysis shared with them and so rarely is data kept in-country – it returns to donor countries’ archives. Luckily this is slowly changing with ‘open data’ and IATI Aid Transparency.

I found that the largest donor that undertook some post project evaluation that actually goes out to the field is the Japanese International Cooperation Agency (JICA), followed by quite a few documents in OECD, and some at the World Bank. What was so surprising was the number of ‘post project’ documents which never returned to interview participants, and most of them are results of desk studies.

My first shock was that no one evaluates and second, even when they evaluate, they don’t go back and talk to anyone.

Some reports such as at the World Bank, say that they have spoken to stakeholders, but don’t provide details as who they are. They could have just spoken to someone in the ministry in the capital. So, the scandalous thing for me is that we do not seem to value the voices of those people whose lives we are here to change to tell us how sustained results were, or not. In the four years I have been working on this, I have come to see how often our ‘global development’ quest is around proving success while we have resources invested in projects there, so that we can document great results to get more resources to feed the aid industry rather than returning to see what capacities we built together that they could sustain themselves. Without post project evaluation, how can we look at the cost effectiveness of our investments?  The other shock for me is that even among the projects evaluated after the project ended, no one ever looks at what emerged from the efforts of the communities and partner organizations themselves, from their own efforts. We only look at what we expect to be there. So if we don’t back, we can’t find out what actually is still there of the project and what they could create themselves.

I started providing post project evaluation primarily because, so few are undertaken and that there is nobody really knows the definitive way to do them. We found that methods used vary widely, from only quantitative surveys to only open-ended questions to a mix of both. Many reports call for post project (ex-post) evaluation, some reports are called post project evaluations but are not actually looking at sustainability of outcomes and impacts. Based on our research, over the last 40 years, far fewer than 1% of all projects have been evaluated. We have found 900 such reports, of which we estimate 600 have field-informed research for post-project results including participant/ partners’ input. We do them rarely and don’t (yet) know how to do them well. Worse yet, very few brave people evaluate anything other than what they already consider not a success, JICA being a notable exception. Two old clients of mine were willing to return to closed projects, so I pulled together teams under Valuing Voices to design what we now call “Sustained and Emerging Impacts Evaluations” (SEIE) using a methodology to looks at not only what we expect and hope to find, explore why or why it was not sustained, but also looks for what emerged which we could not possibly expect. But while we advocate for it widely and many evaluators acknowledge how few are done, there are few people banging on our door!

One of the things that I found most distressing was when I did a post project evaluation in a country for a client and the results were very bad. Around 10 per cent of the project outcomes remained and while there were quite a few external reasons for it, they were unwilling to share the findings within their own organization, much less outside. Needless to say, we could not publish this either.

They were afraid that if people find out that things were not sustained, they would assume they were not done well, and that they had wasted donor resources and then would not get more funding and possibly they would lose their jobs.

Yet the demand from our partners is high!  During our local debrief, a government official who is 80 years old said “I have been working with donors who have been coming to my country for sixty years, but no one has ever come back to tell us the results of an evaluation and much less to come back when a project closed and look at the sustainability”. He asked, “Can you get us funding? We are happy to do more of these evaluations.”

I sat there and I thought that for sixty years he has been the person in the government who needed to hear what worked or didn’t, and no one has ever come back. This speaks to the fact that we don’t seem design with them, we do it for them.  Locals have asked me “how can we sustain something we did not design?”. Evaluation can be a check the box-the-money-is-spent-lets-move-on.

Unfortunately, we have lost sight that our true clients are the participants and partners themselves; now, implementers’ true clients have become the donors.

Evaluation post project urges us to think about ‘did we design for sustainability?’, ‘how do we involve participating partners in helping us evaluate results?’, how can we communicate the results and how do undertake knowledge management so that people know what sustained and can build on it?’ (especially vital for those coming later to be aware of projects there previously).

Can you provide examples that can help in improving post project evaluation?

One of the works on lessons from exit strategies was a four-country 2015 study by Professor B. Rogers of Tufts University on Food for Peace/ USAID.  She did a study that said you have to return and explore how do we exit projects better at the end, how do we exit ensuring the sustainability of a project as collectively as we can. They returned to Kenya, Bolivia, India and Honduras 2-3 years after closeout and the results actually changed the strategy of Food for Peace policy. So now, there are pilot projects that are funding post project evaluation for two years after the project ends. It changed how the donor allocates money for projects. This is now being threatened by funding cuts to USAID in the new administration.

How can the development sector address the concerns you raised?

I was in a group of US policy lobbyists who advocate for foreign aid and when we told them that over 1.6 trillion dollars of aid since the year 2000 is hugely unevaluated, that far less than one per cent has been examined post project; these lobbyists who have been working on Capitol Hill for the last 30 years said “you can’t tell anyone this as this will eliminate foreign aid”. I said “I have to tell someone because we could be wasting millions of dollars on projects and not learning what was sustained to do more of that rather than other projects that are not sustainable. How do we make aid more accountable?”

We need to take sustained impacts into consideration. We need to design projects with country nationals and also learn what emerged from their own efforts post project that was unexpected. Donors have to stop thinking themselves on a mission for their own success, they need to recognize that we are meddling in people’s lives who need to be involved at all stages to sustain successful results for their lives. Otherwise we are throwing hundreds of millions of dollars out the window. A handful of projects have been evaluated and even when I emailed JICA, who have done most of the post project evaluations, I asked them to tell me more about how do they use the results but they never answered.

I am rather hopeful about the growth of impact investing and Corporate Social Responsibility (CSR) investments. Some of our lessons can help shape projects they invest in, the data they track to assess the likely sustainability and participation of locals in the projects. Such investors have a longer timeframe because the core of the problem with the donors is the fixed project cycle of 1, 3, 5 years. Private sector investors can build in the luxury of continuing projects based on results data, or closing them, rather than fixed implementation.  We need to stop wasting people’s time, resources, hope.

How will we perform on the ambitious SDGs? 

The SDGs say that we are going to eliminate poverty, make people healthy and food secure; which are amazing aspirations. The private sector has gotten hold of them and are trying to use their investments in these areas. This is marvellous effort.

What is the disconnect is that while we are tracking income increasing, health improving, etc, but there is no exploration of what is causing this. The data can be showing results that is is improving’ , and we are assuming that the project investments are causing the increase and maybe they are, or maybe it is just that the economy is getting better and this is disconnected from projects. Once we leave, will this improvement continue to sustain?  I am in one of the SDG working groups and I asked, “will your country track down if the results have gone down and is it linked to results of any project funding disappearing?” They said ‘maybe’; it is not part of our plan. We must report on indicators.  There is no feedback loop whether aid helps or harms. And if it is helping, where it is helping more than elsewhere so there is much to be explored.

For SDGs, the task of reporting is given to countries without much funding, and tracking some of the 232 indicators is complex.

Our research has unearthed the obvious – that people sustain what helps them the most. If an agriculture project is feeding their family better, then they will continue to do that. Other project activities like planting trees could stop unless they provide long-term and visible benefits directly to the people.  If sustainability is a goal, then it changes design.  When a project ends, and if we see that no more trees are planted, then why continue to invest our time and money funding more tree-planting with the expectation people will continue with their own paltry resources. Data teaches us.

We discussed the problems of traditional donors that have been mostly from Western countries. Now you see China, what is your expectation from the new donors?

Part of me feels they will value their investments as they have been themselves aid recipient countries. They may value the voices of their partners and participants more than what was done previously. But we have to look in details. The idea of transparency and accountability as a concept are very much a western concept and I am hopeful that they will take this aspect into considering sharing lessons from investing in different countries.

I am also hoping that the impact investor and CSR approach will take off on a large scale with the right intentions: to both profit and help. The challenge is that private investors will invest only in a narrow band of successful projects. We are talking about their replacing the full band of development aid but there are major issues around humanitarian aid for the poorest, most marginalized – who will help them?

If you are interested in knowing more about Valuing Voices, click here. They are also available on Twitter: @WhatWeValue

There are many who are concerned or have faced uncomfortable or inspiring situations during their work on sustainable development. If you want to share your experiences , please write to us at contact@sixdegreesnews.org.