Reflections on the Changing Needs of M&E

I’ve recently completed a number of missions for M&E purposes across Asia and the Pacific, and although there is always a sense of satisfaction of ticking the box on all of your deliverables, of knowing your client is pleased with the results, there is perhaps a greater satisfaction in what you can take away from each experience to build your own knowledge and try to use in your next assignment, to make development work just a bit more accountable and a bit… more.

Measuring Innovation in Laos

Nearly 10 years after drafting guidelines for the country office of an international organization on how to implement results-based monitoring and evaluation, I’ve discovered that a) not only have those guidelines been disseminated and used regionally, but b) they haven’t been updated (10 years!!??) and c) we’ve changed what we want to measure but not the tools we are allowed to measure the changes we effect with. Sigh.

Case in point. Two weeks ago, in desperation, I shoe-horned outcome mapping into a log frame. Like some sort of M&E superhero (ok, stretching that definition a bit, but it felt like a heroic effort nonetheless). But I literally couldn’t stomach, as a responsible professional, the idea that we would somehow be able to effectively measure a) policy dialogue and b) policy innovation through numbers alone. Then I had to gear up to argue the following point: when it comes to policy dialogue outcomes, and supporting innovation, we simply cannot set targets (gasp!). That whole section of the log frame is going to have to remain blank. Why? If we identify targets we want to achieve, then we’re not being innovative. The whole point of being innovative is to ‘see what happens.’ And then learn.

Surprisingly, everyone agreed. They were hesitant about not having a fixed target to report against (which was solved by the idea of developing X number of lessons learned reports at certain points during the project to highlight what worked, what didn’t, where we can learn, and use it as a tool to support government policy making. Nice). But everyone understood, as soon as we set a specific numerical or policy target, it’s no longer innovative. It’s no longer driven by ideas from government and civil society, but by international good practice and development requirements that we measure everything. To death.

The big lesson for me is that we, as development practitioners, talk a big game about innovation but then immediately stifle that innovation because we need to know – in advance – what the outcome will be. We have literally log-framed ourselves into a corner that we can’t escape from because we have conditioned ourselves to think that the only success that counts is that which we have predicted.

The Effects of ‘Siloed’ Development in Cambodia

Once upon a time, there was a lot of talk about ‘breaking down silos’ in development. I’m sure that there probably continues to be, and there must be a new buzzword, but I don’t know what it is. Nonetheless, as with many ideas, it was, in my experience, all talk and no real action. The occasional reference of cooperation between projects, occasional informal information sharing between staff, perhaps a joint workshop. But nothing that really linked the work of one sector to another. And in my observation, no one really worried too much about this, myself included. I mean, obviously well-coordinated project implementation across sectors would be fantastic, but if we didn’t do it, what was the real harm (except accusations of inefficiency)?

Like anything, for me, this was a case of ‘see it to believe it’ when it became obvious that poorly coordinated development projects can indeed lead to more harm (in some respects) than good.

In Cambodia, I was evaluating a project on mine-action – mine clearance and land release to be specific – and as anyone can imagine, it seemed this could only have a positive outcome (aside from inevitable delays in implementation, etc). On the one hand, of course it was positive – cleared and released land means safe land. Agriculture can recommence. Children can play football in the fields. People can safely get to school, or the doctor or the market. Yes, indeed, success story.

But the government was keen to demonstrate that mine action lead directly to local development. The project staff were quick to point out (and caution) that there is no causal link between safe land and development, but wishful thinking and all that. So part of my assignment was to find something – anything – to suggest mine action was leading to reductions in poverty. I assumed there would be evidence of something (more children in school? More local market vendors? Increased HH income?) so imagine my shock to discover that, at the household level, poverty was increasing and more people were migrating to look for work.

The situation is more complex than I have time or space to lay out here. Suffice to say that there was a pervasive belief that once agriculture land was cleared, everyone could plant their crops and then make money at harvest time. But no one came around to talk to people who own small plots of land about the likelihood that they would invest more than they would get back, that the land was too small to produce the volume necessary to make it a worthwhile enterprise. Households were getting informal microloans to buy seed and fertilizer and machinery, and getting so little in return due to low commodity prices and high interest rates on their loans that they were going further into debt. More and more people were heading back to Thailand to work as labourers, and in some cases whole families were selling up and fleeing their debts. Among those who stayed, they just kept plowing more money back into a losing enterprise because no one told them not to (pun intended).

It was so obvious what the problem was. Land was being cleared and people encouraged to get back to farming, but no one told them that farming has changed. Small plots are not sufficient to meet household income needs, let alone break out of the poverty trap. And for all of the many development projects working on improving agriculture and economic development – where were they? Not in these villages. They were in areas where the land was already safe, where agriculture was already productive. Where they needed to be was in the villages where agriculture was just starting back up after decades of disuse due to the presence of land mines.

When I presented my findings to the government and donors, there was initial disbelief – until I laid out the math. The loan, the investments, the income. It was not a happy balance sheet. To be fair, this was not a black mark on the project – the objective of the project was to make land safe. Done. But somehow, there needs to be follow through, and not just at the end of the project. One donor noted that ‘really, we need to be more responsible that this. We need our agriculture projects to sweep in after the land is cleared to make sure this doesn’t happen.’

We can’t claim to have done no harm when people are safer, but also poorer.

In effect, while no one project can solve all problems, and inevitably solving one problem means uncovering many more. This case is an important learning opportunity – development is complex and we can no longer operate under the illusion that lack of coordination and cooperation hurts no one. We need to ask ourselves ‘so we do this project, and then what? What support will be needed next?’

And then we get over ourselves – and our institutional egos – and ask the guy in the next office or the building across the road ‘do you think you can help?’

Understanding Safeguards vs Social Responsibility in Samoa

Like many people in the development and humanitarian field, Samoa was a big tick on the bucket list. Not because of the beauty of the island (one of the loveliest), or the friendliness of the people (in the top five for sure), but because there was a cyclone. I’ve had earthquakes (many), flooding (often), landslides (terrible), war and unrest (devastating) and a few tropical storms. But this was my first proper cyclone and it was enlightening. First, because you can prepare for it, you actually do (candles, water, crisps, biscuits, wine, of course). Top up the credit on the phone. The hotel management reinforced the windows. And then you lay in bed and feel the wind buffeting the building (on the ground floor no less) and wonder how on earth people survive Category 5 storms? This was a Category 2-3 when it hit Samoa (and a Category 5 by the time devastated Tonga), and the damage was extensive, although there were, to my knowledge, no deaths. The next day finds you standing on the sea wall waving your phone around trying to get a signal strong enough and long enough to send a message to your husband that you’re fine (but you’ve have to change rooms because yours ended up with too much water damage to be healthy). Then you head out to check out the damage and think: note to self, bring better shoes next time. These ballet flats are useless with all of this mud. (There was A LOT of mud after the water receded).

And by some twist of fate, this experience fed directly into the work I was doing – revising an M&E framework on ecosystem-based livelihoods to be more gender-sensitive and socially responsible. While on my little walkabout, I chatted with shopkeepers who were cleaning up after being inundated with mud and water. A lot of damage, a lot of lost merchandise, and no insurance. Because these are small independent shops and no one expects that a tropical storm will turn into a Category 3 cyclone in the space of a few hours – overnight – no one expected, or prepared for, the flooding to the extent which happened.

And a little thought began to wiggle around in my head that in the M&E framework we were missing something – and by something I mean something critical – and I went back to my hotel and spent six hours pulling it apart and reviewing the project document, again and again, until I realized what was wrong.

The project – a part of it – was introducing micro-financing for ecosystem-based livelihoods that would contribute to climate adaptation and the local economy. Sounds snazzy and very trendy. Made a lot of sense, too, as Samoa tries to push back against imported foods and advocate for diets focused on what is grown locally. Win-win. Except for the part where we confuse social safeguards with social accountability.

The project planned to put a lot of safeguards in place to reduce to the extent possible the impacts of the infrastructure work being done on local livelihoods. There were the complaint mechanisms. And affirmative action to ensure that the poorest and most vulnerable were able to capitalize on cash-for-work schemes and micro-finance opportunities. But there was nothing about insurance.

What happens when another cyclone descends and destroys the banana crop or the coconut trees and the family which was encouraged to take out a micro-finance loan as part of the project can’t repay it because they’ve lost their capital? In Samoa, even if your crop isn’t lost to the wind, it could be lost to a landslide. Or the devastation of an earthquake. What then? What happens when families are pushed further into poverty because we encouraged them to take risks without ensuring they had the tools (ie: insurance) to mitigate the likelihood of those risks?

In the office the next day, I brought this up with the team. They all looked at me with wide eyes and – to their credit – immediate understanding. Work is ongoing to ensure that some sort of risk management scheme is included in the micro-finance activities.

This type of lesson doesn’t fit neatly within M&E, but it is a lesson that we need to take onboard when we are designing indicators around social protection and responsibility towards project beneficiaries. It’s one more aspect of ‘do no harm’ that helps us understand that socially-responsible development means that we recognize that we cannot guarantee that a project will ‘do no harm’, but we must do our utmost to help beneficiaries manage the risks that we have passed on to them along the way.